# nlp **URL:** https://discuss.pytorch.org/c/nlp/8.md?page=3 [Latest](https://discuss.pytorch.org/latest.md) · [Categories](https://discuss.pytorch.org/categories.md) **Page:** 4 --- ## [Flex Attention Extremely Slow](https://discuss.pytorch.org/t/flex-attention-extremely-slow/209724)
**Author:** [@whywhy](https://discuss.pytorch.org/u/whywhy)\ **Replies:** 1\ **Last updated:** [September 20, 2024, 12:23am UTC](https://discuss.pytorch.org/t/flex-attention-extremely-slow/209724 "2024-09-20T00:23:28Z")
I try to use flex attention in huggingface transformer, only to find it very slow. Compared to the sdpa implementation, flex attention is about 4-5 times slower, but it does save the CUDA memory. Here is the example cod… --- ## [How tokens per second calculated for LLM training](https://discuss.pytorch.org/t/how-tokens-per-second-calculated-for-llm-training/209728)
**Author:** [@bigtree](https://discuss.pytorch.org/u/bigtree)\ **Replies:** 0\ **Last updated:** [September 18, 2024, 5:45am UTC](https://discuss.pytorch.org/t/how-tokens-per-second-calculated-for-llm-training/209728 "2024-09-18T05:45:16Z")
how are tokens per second calculated for LLM training? Is this the same way as for LLM inference? --- ## [Drop row from tensor in cuda](https://discuss.pytorch.org/t/drop-row-from-tensor-in-cuda/209489)
**Author:** [@Augusto\_Maillo](https://discuss.pytorch.org/u/Augusto_Maillo)\ **Replies:** 3\ **Last updated:** [September 14, 2024, 2:31pm UTC](https://discuss.pytorch.org/t/drop-row-from-tensor-in-cuda/209489 "2024-09-14T14:31:10Z")
I’m using Whisper by transformers library to transcribe some audios. In my application I need to transcribe near real time, so I use batching to speed up. However whisper generation is sequential, and some samples in bat… --- ## [Unhashable list while training sbert](https://discuss.pytorch.org/t/unhashable-list-while-training-sbert/209563)
**Author:** [@mantics](https://discuss.pytorch.org/u/mantics)\ **Replies:** 0\ **Last updated:** [September 14, 2024, 12:44pm UTC](https://discuss.pytorch.org/t/unhashable-list-while-training-sbert/209563 "2024-09-14T12:44:57Z")
Hello, I am trying to create embeddings first before feeding into the model. Why I need to create the embeddings first is because of max\_seq\_len, I want to make use of long sequences and I think the best way is to gener… --- ## [RuntimeError: CUDA error: device-side assert triggered, LayoutLM Fine-Tuning](https://discuss.pytorch.org/t/runtimeerror-cuda-error-device-side-assert-triggered-layoutlm-fine-tuning/185293)
**Author:** [@JIMUEL\_SERVANDIL](https://discuss.pytorch.org/u/JIMUEL_SERVANDIL)\ **Replies:** 10\ **Last updated:** [September 10, 2024, 2:11pm UTC](https://discuss.pytorch.org/t/runtimeerror-cuda-error-device-side-assert-triggered-layoutlm-fine-tuning/185293 "2024-09-10T14:11:12Z")
I am trying to run LayoutLM, with the following arguments: ! CUDA\_LAUNCH\_BLOCKING=1 python run\_seq\_labeling.py \\ --data\_dir "{dataset\_dir}" \\ --labels "{label\_fil… --- ## [Model predicted almost correct sentences at the time of training but is only predicting \ token at the time of test](https://discuss.pytorch.org/t/model-predicted-almost-correct-sentences-at-the-time-of-training-but-is-only-predicting-start-token-at-the-time-of-test/209353)
**Author:** [@Ansh\_Dadhich](https://discuss.pytorch.org/u/Ansh_Dadhich)\ **Replies:** 0\ **Last updated:** [September 10, 2024, 9:30am UTC](https://discuss.pytorch.org/t/model-predicted-almost-correct-sentences-at-the-time-of-training-but-is-only-predicting-start-token-at-the-time-of-test/209353 "2024-09-10T09:30:28Z")
loss was like loss at epoch 61 is 0.01835295371711254 \[2, 3128, 4624, 4455, 3830, 3524, 4574, 3387, 2814, 2971, 4815, 3, 2729, 3551, 2882, 2893, 4405\] \<\< START \>\> me too lets forget about germs and focus on food loss… --- ## [Self Self-attention implementation results are 'a bit' suprising](https://discuss.pytorch.org/t/self-self-attention-implementation-results-are-a-bit-suprising/209342)
**Author:** [@dey](https://discuss.pytorch.org/u/dey)\ **Replies:** 0\ **Last updated:** [September 10, 2024, 7:20am UTC](https://discuss.pytorch.org/t/self-self-attention-implementation-results-are-a-bit-suprising/209342 "2024-09-10T07:20:31Z")
I am studying attention so I was making some experiments on implementing. I was curious of each part’s (Residual connections, Layer normalization, Dropout, Multi-heads) importance on training so I tried to implement the… --- ## [Extracting embeddings from log probabilities](https://discuss.pytorch.org/t/extracting-embeddings-from-log-probabilities/209330)
**Author:** [@Debodeep\_Banerjee](https://discuss.pytorch.org/u/Debodeep_Banerjee)\ **Replies:** 0\ **Last updated:** [September 9, 2024, 10:03pm UTC](https://discuss.pytorch.org/t/extracting-embeddings-from-log-probabilities/209330 "2024-09-09T22:03:32Z")
Hi, I am solving an image captioning related issue and eventually I have extract the embeddings of the tokens. One possible way is to extract the embeddings using the tokens. But I cannot do do that because in my case th… --- ## [Can transformer automatically learn the length of sequences?](https://discuss.pytorch.org/t/can-transformer-automatically-learn-the-length-of-sequences/209281)
**Author:** [@JimW](https://discuss.pytorch.org/u/JimW)\ **Replies:** 0\ **Last updated:** [September 9, 2024, 6:18am UTC](https://discuss.pytorch.org/t/can-transformer-automatically-learn-the-length-of-sequences/209281 "2024-09-09T06:18:50Z")
I am new to transformers and I need to use transformer to predict a score based on a input sequence of variable length. Due to the nature of my use case, longer sequnences tend to provide higher score for regression, al… --- ## [Finen tuning Llama with using pytorch in colab](https://discuss.pytorch.org/t/finen-tuning-llama-with-using-pytorch-in-colab/208811)
**Author:** [@merto58](https://discuss.pytorch.org/u/merto58)\ **Replies:** 1\ **Last updated:** [August 29, 2024, 6:23pm UTC](https://discuss.pytorch.org/t/finen-tuning-llama-with-using-pytorch-in-colab/208811 "2024-08-29T18:23:26Z")
Hello. Im having an issue with fine tuning Llama 3.1. Google cloab crashes whenever I try to train on my custom dataset. I think its about pytorch and tensorflow versions. Im tried to use gpu and tpu both gave me error.… --- ## [Output.loss is None when training model](https://discuss.pytorch.org/t/output-loss-is-none-when-training-model/208668)
**Author:** [@DanielTobi0](https://discuss.pytorch.org/u/DanielTobi0)\ **Replies:** 0\ **Last updated:** [August 26, 2024, 3:43pm UTC](https://discuss.pytorch.org/t/output-loss-is-none-when-training-model/208668 "2024-08-26T15:43:02Z")
from datasets import load\_dataset from transformers import AutoTokenizer, AutoModelForCausalLM, AdamW, get\_scheduler from torch.utils.data import DataLoader, Dataset import torch # Load dataset dataset = load\_dataset("m… --- ## [HELP with multilabel classification and BCEWithLogitsLoss](https://discuss.pytorch.org/t/help-with-multilabel-classification-and-bcewithlogitsloss/208094)
**Author:** [@Victor\_Jansen](https://discuss.pytorch.org/u/Victor_Jansen)\ **Replies:** 1\ **Last updated:** [August 14, 2024, 9:48pm UTC](https://discuss.pytorch.org/t/help-with-multilabel-classification-and-bcewithlogitsloss/208094 "2024-08-14T21:48:16Z")
Im trying to classificate multilabels for sentiment analysis. first i separate the data and pass the batch on BERT layer, that encodes and generate embedding. So i create a NN that uses this data from BERT to classify … --- ## [Torchtext not supported](https://discuss.pytorch.org/t/torchtext-not-supported/207715)
**Author:** [@SatyaDasika](https://discuss.pytorch.org/u/SatyaDasika)\ **Replies:** 3\ **Last updated:** [August 13, 2024, 10:58pm UTC](https://discuss.pytorch.org/t/torchtext-not-supported/207715 "2024-08-13T22:58:33Z")
If torchtext is no longer supported. is there an alternative for the same with pytorch 2.0 --- ## [NLP indexing question](https://discuss.pytorch.org/t/nlp-indexing-question/207942)
**Author:** [@Nima\_Chelongar](https://discuss.pytorch.org/u/Nima_Chelongar)\ **Replies:** 3\ **Last updated:** [August 13, 2024, 2:34pm UTC](https://discuss.pytorch.org/t/nlp-indexing-question/207942 "2024-08-13T14:34:15Z")
hi guys. i want to train LSTM network for filling a form using a given text. now when i turn my inputs and labels to tokens, i have something like: tensor(\[\[ 0, 15, 14, ..., 0, 0, 0\], \[ 0, 15, 14, ..., 0… --- ## [Next step after NLP specialization](https://discuss.pytorch.org/t/next-step-after-nlp-specialization/207492)
**Author:** [@Abdulrahman\_Ahmed](https://discuss.pytorch.org/u/Abdulrahman_Ahmed)\ **Replies:** 1\ **Last updated:** [August 12, 2024, 2:37am UTC](https://discuss.pytorch.org/t/next-step-after-nlp-specialization/207492 "2024-08-12T02:37:09Z")
I’m about to finish the last course in NLP specialization on Coursera, i want to know the next step in the study to be ready for work? --- ## [Integrated gradients with captum and handmade transformer model](https://discuss.pytorch.org/t/integrated-gradients-with-captum-and-handmade-transformer-model/180407)
**Author:** [@francesco\_patane](https://discuss.pytorch.org/u/francesco_patane)\ **Replies:** 8\ **Last updated:** [August 9, 2024, 12:38am UTC](https://discuss.pytorch.org/t/integrated-gradients-with-captum-and-handmade-transformer-model/180407 "2024-08-09T00:38:56Z")
hi! i’m using captum with a transformer based protein language model in order to identify input (embeddings)-output correlations. i take inspiration from captum website tutorials (BERT model) but i’m not able to run last… --- ## [Custom Model with 2 GPT2 models from huggingface](https://discuss.pytorch.org/t/custom-model-with-2-gpt2-models-from-huggingface/207681)
**Author:** [@Yousaf\_Raza](https://discuss.pytorch.org/u/Yousaf_Raza)\ **Replies:** 0\ **Last updated:** [August 7, 2024, 7:07am UTC](https://discuss.pytorch.org/t/custom-model-with-2-gpt2-models-from-huggingface/207681 "2024-08-07T07:07:12Z")
I want to make a custom model. Custom model will include 2 GPT2 models but following modifications are needed at each attention layer. self attention outputs of adjacent layers will get cross attended and cross attentio… --- ## [SDPA backend routes requirement](https://discuss.pytorch.org/t/sdpa-backend-routes-requirement/207660)
**Author:** [@Sohaib\_Ahmed](https://discuss.pytorch.org/u/Sohaib_Ahmed)\ **Replies:** 0\ **Last updated:** [August 6, 2024, 9:53pm UTC](https://discuss.pytorch.org/t/sdpa-backend-routes-requirement/207660 "2024-08-06T21:53:29Z")
What are the input requirements that tells SDPA to use memory-efficient, flash-attention or math backend. I saw some repo to use is\_causal=True instead of attetnion\_mask in order to dispatch to flash-attention backend bu… --- ## [I'm trying to build up a rag\_chain, but encountering this error——TypeError: embedding(): argument 'indices' (position 2) must be Tensor, not ChatPromptValue](https://discuss.pytorch.org/t/im-trying-to-build-up-a-rag-chain-but-encountering-this-error-typeerror-embedding-argument-indices-position-2-must-be-tensor-not-chatpromptvalue/207550)
**Author:** [@P1sc3s007](https://discuss.pytorch.org/u/P1sc3s007)\ **Replies:** 1\ **Last updated:** [August 5, 2024, 8:47pm UTC](https://discuss.pytorch.org/t/im-trying-to-build-up-a-rag-chain-but-encountering-this-error-typeerror-embedding-argument-indices-position-2-must-be-tensor-not-chatpromptvalue/207550 "2024-08-05T20:47:52Z")
from transformers import AutoTokenizer, AutoModelForCausalLM import torch from langchain\_community.document\_loaders import TextLoader # 文本加载器 from langchain.text\_splitter import CharacterTextSplitter # 文本分块器 import weav… --- ## [Sizes of tensors must match except in dimension 1](https://discuss.pytorch.org/t/sizes-of-tensors-must-match-except-in-dimension-1/147785)
**Author:** [@Gorgen](https://discuss.pytorch.org/u/Gorgen)\ **Replies:** 6\ **Last updated:** [August 3, 2024, 10:26pm UTC](https://discuss.pytorch.org/t/sizes-of-tensors-must-match-except-in-dimension-1/147785 "2024-08-03T22:26:09Z")
Dear everyone: when i train the model. it reported Expected size 7 but got size 5 for tensor number 1 in the list. I don’t know what expected object is needed to be 7. Could you please help me to fix this issue? Thank… --- ## [Multi-task learning: Bottleneck, multi-GPU](https://discuss.pytorch.org/t/multi-task-learning-bottleneck-multi-gpu/207322)
**Author:** [@Tai-Mai](https://discuss.pytorch.org/u/Tai-Mai)\ **Replies:** 0\ **Last updated:** [July 31, 2024, 2:47pm UTC](https://discuss.pytorch.org/t/multi-task-learning-bottleneck-multi-gpu/207322 "2024-07-31T14:47:53Z")
I wanted to explore the following model architecture: A Llama-2 model that has two different task heads: One for classification and the other for causal language modeling. My goal is to train one model that can take a t… --- ## [Different value between BertSDPA scratch and pretrained sequence classification](https://discuss.pytorch.org/t/different-value-between-bertsdpa-scratch-and-pretrained-sequence-classification/207272)
**Author:** [@jeyan9](https://discuss.pytorch.org/u/jeyan9)\ **Replies:** 0\ **Last updated:** [July 30, 2024, 7:49pm UTC](https://discuss.pytorch.org/t/different-value-between-bertsdpa-scratch-and-pretrained-sequence-classification/207272 "2024-07-30T19:49:26Z")
Hello, I am using the bert pretrained parameter as an input to accelerator. But the problem is that the result of BertSdpaSelfAttention in BertForSequenceClassifcation gives different output value with the torch F.scaled… --- ## [How do I apply Batch Normalization on a sequential data?](https://discuss.pytorch.org/t/how-do-i-apply-batch-normalization-on-a-sequential-data/102106)
**Author:** [@rectified\_sigmoid](https://discuss.pytorch.org/u/rectified_sigmoid)\ **Replies:** 5\ **Last updated:** [July 28, 2024, 10:03am UTC](https://discuss.pytorch.org/t/how-do-i-apply-batch-normalization-on-a-sequential-data/102106 "2024-07-28T10:03:00Z")
I am trying to train an NLP model that takes in the entire sequence at once instead of passing each time step individually as this approach is faster afaik. So, my data is of shape (seq\_size, batch\_size, length). This fo… --- ## [OpenNMT beam.py beam search class for seq2seq language models](https://discuss.pytorch.org/t/opennmt-beam-py-beam-search-class-for-seq2seq-language-models/206932)
**Author:** [@MLangner](https://discuss.pytorch.org/u/MLangner)\ **Replies:** 0\ **Last updated:** [July 25, 2024, 8:04am UTC](https://discuss.pytorch.org/t/opennmt-beam-py-beam-search-class-for-seq2seq-language-models/206932 "2024-07-25T08:04:26Z")
I am trying to reuse code from the slightly outdated openNMT library for beam search in a custom language model (OpenNMT-py/onmt/translate/beam.py at 49185121e46c4f65d68101590bad231a8dd73e4f · OpenNMT/OpenNMT-py · GitHub)… --- ## [Which is better TorchServe Batching or ClientSide Batching for text classification](https://discuss.pytorch.org/t/which-is-better-torchserve-batching-or-clientside-batching-for-text-classification/206856)
**Author:** [@OmniaZayed](https://discuss.pytorch.org/u/OmniaZayed)\ **Replies:** 0\ **Last updated:** [July 24, 2024, 11:57am UTC](https://discuss.pytorch.org/t/which-is-better-torchserve-batching-or-clientside-batching-for-text-classification/206856 "2024-07-24T11:57:11Z")
Hi, I am working on a text classification project, and I am researching the best way to do batching if I want to send a csv file to the model to classify at once. I couldn’t find examples that show batching for text, b… --- ## [Discrepancy Between key\_padding\_mask and attn\_mask in MultiheadAttention Layer](https://discuss.pytorch.org/t/discrepancy-between-key-padding-mask-and-attn-mask-in-multiheadattention-layer/203322)
**Author:** [@evaggelos](https://discuss.pytorch.org/u/evaggelos)\ **Replies:** 9\ **Last updated:** [July 23, 2024, 1:14pm UTC](https://discuss.pytorch.org/t/discrepancy-between-key-padding-mask-and-attn-mask-in-multiheadattention-layer/203322 "2024-07-23T13:14:32Z")
Hello, I am working with the MultiheadAttention layer in PyTorch and encountered a discrepancy between using key\_padding\_mask and attn\_mask for handling variable length sequences with padding. My goal is to ensure that … --- ## [Have a question on \`W\_ii @ x\_t\` in lstm](https://discuss.pytorch.org/t/have-a-question-on-w-ii-x-t-in-lstm/206598)
**Author:** [@Seungjun\_Lee](https://discuss.pytorch.org/u/Seungjun_Lee)\ **Replies:** 0\ **Last updated:** [July 20, 2024, 3:35am UTC](https://discuss.pytorch.org/t/have-a-question-on-w-ii-x-t-in-lstm/206598 "2024-07-20T03:35:18Z")
I have a question of the shape of weights of lstm. If the input to the lstm layer is (batch\_size, seq\_length, input\_size), then x\_t would be (batch\_size, input\_size). And as the documentation say so, the shape of W\_ii … --- ## [nn.TransformerEncoder for classification](https://discuss.pytorch.org/t/nn-transformerencoder-for-classification/83021)
**Author:** [@n8henrie](https://discuss.pytorch.org/u/n8henrie)\ **Replies:** 9\ **Last updated:** [July 20, 2024, 1:39am UTC](https://discuss.pytorch.org/t/nn-transformerencoder-for-classification/83021 "2024-07-20T01:39:19Z")
Hello all, I’m trying to get the built-in pytorch TransformerEncoder to do a classification task; my eventual goal is to replicate the ToBERT model from this paper (paperswithcode is empty). Unfortunately, my model does… --- ## [How to reduce TPU RAM usage when calculating large tensors](https://discuss.pytorch.org/t/how-to-reduce-tpu-ram-usage-when-calculating-large-tensors/201895)
**Author:** [@Acerodancisto](https://discuss.pytorch.org/u/Acerodancisto)\ **Replies:** 1\ **Last updated:** [July 19, 2024, 12:23pm UTC](https://discuss.pytorch.org/t/how-to-reduce-tpu-ram-usage-when-calculating-large-tensors/201895 "2024-07-19T12:23:19Z")
I am currently working on dealing with the task vector of the llama’s parameters, which is huge (over 13015864320). And I have a tensor shape (3,13015864320) in float16. Every time I try to get kthvalue or do element wis… --- ## [Can someone help me understand how to use a .pt file?](https://discuss.pytorch.org/t/can-someone-help-me-understand-how-to-use-a-pt-file/206345)
**Author:** [@anonymous\_struggler](https://discuss.pytorch.org/u/anonymous_struggler)\ **Replies:** 2\ **Last updated:** [July 16, 2024, 8:14am UTC](https://discuss.pytorch.org/t/can-someone-help-me-understand-how-to-use-a-pt-file/206345 "2024-07-16T08:14:29Z")
I am really new to this field and have a few questions. I am trying to quantize a model using MIT Han lab AWQ quantization method. I have successfully pulled the model from huggingface and applied quantization inside a d… [Previous page](https://discuss.pytorch.org/c/nlp/8.md?page=2) [Next page](https://discuss.pytorch.org/c/nlp/8.md?page=4)