Imran Nazir

AI Applied Engineer · LLM Serving · GPU Inference Infrastructure

I build and operate the infrastructure that serves LLMs in production, and work upstream on the inference stacks that run them.

About

I design and operate the infrastructure that serves LLMs at production scale — currently supporting 50M+ users as an AI Applied Engineer at QuillBot (Learneo). My work sits at the intersection of GPU inference internals, distributed systems, and ML platform engineering.

Trained originally as a civil engineer (B.Tech, BIT Sindri, 2023), I moved into ML systems through self-directed study and have since focused on the operational side of AI: making inference fast, reliable, and cheap at scale.

Research & Focus Areas

• LLM serving at scale — throughput/latency trade-offs across vLLM, SGLang, and Triton Inference Server
• GPU inference internals — batching strategies, KV-cache management, capacity-aware scheduling
• ML systems architecture — designing platforms that stay reliable under production load
• Kubernetes-native orchestration for asynchronous, high-throughput inference workloads

Open Source

	Hugging Face · transformers Attention correctness · SDPA · seq2seq training Decoder causality fix · root-cause trace · regression tests View work →
	OpenNMT · CTranslate2 Inference runtime · model conversion · C++ UMT5 support · per-layer attention bias · validated on a 2.9B checkpoint View work →

Selected upstream impact — PR evidence

Project	Contribution	Product / user impact
transformers	#49135 — UMT5 decoder self-attention was not causal under SDPA	Restored causality for every UMT5 model trained or evaluated through the default Seq2SeqTrainer path; position-0 logit drift fell from ~1.03e-1 to ~1.91e-6
CTranslate2	#2103 — UMT5 inference support via per-layer relative attention bias	Made UMT5 servable on CTranslate2 for the first time, closing a 2+ year feature request with byte-identical output for existing T5/mT5 conversions

transformers #49135 — fix UMT5 decoder self-attention not being causal

• Problem: UMT5LayerSelfAttention never passed is_causal to UMT5Attention. Under SDPA the causal mask is skipped when there is no padding, so during full-sequence teacher forcing — the path DataCollatorForSeq2Seq and Seq2SeqTrainer produce by default — the decoder could attend to future positions. generate() was unaffected, since it decodes one position at a time.
• Root cause: introduced in #47014, which moved the T5 family to SDPA and added the is_causal parameter without updating the call sites. v5.14.1 was correct; v5.15.0 onward was not.
• Fix: pass is_causal=config.is_decoder as T5/MT5 already do, and set is_causal=False explicitly on cross-attention. Added regression tests for the is_causal flag across all three attention types and for position-0 logit stability.
• Verified: log-probability drift at position 0 dropped from ~1.03e-1 to ~1.91e-6, matching the last known-good release.
• Merged into main; filed as #49134.

CTranslate2 #2103 — add UMT5 support (per-layer relative attention bias)

• Problem: CTranslate2 could not convert UMT5. Unlike T5/mT5, which compute a relative attention bias at layer 0 and share it, UMT5 gives every self-attention layer its own bias table — a layout neither the converter nor the runtime stacks supported.
• Fix: added a UMT5Loader that preserves per-layer bias tables and overrides get_vocabulary so <extra_id_*> sentinels are not duplicated, and taught the encoder/decoder stacks to select a per-layer bias path only for models that need it — detected by pointer identity on aliased variables, so no config flag or model-format change was required.
• Context: closed a 2+ year-old request (#1478) that had stalled because the obvious fix risked slowing existing T5/mT5 models. t5-small model.bin is byte-identical before and after the converter change.
• Verified: added runtime regression tests using distinct, non-constant bias tables per stack, which fail when the runtime is forced to share layer 0's bias; validated end-to-end on a 2.9B-parameter UMT5 checkpoint on an NVIDIA L4.
• Merged into master. The parity gap that surfaced during review is what led to the transformers fix above.

Current footprint: 2 merged PRs across 2 upstream inference projects — one correctness fix, one new model family — both traced from a single numerical discrepancy. I care about landing changes maintainers can trust more than PR volume.

How I Contribute

Root cause over symptom Trace a numerical mismatch to the line that causes it, even across repos.	Evidence before claims Quantify the before/after and pin the first bad commit.
Narrow, reversible diffs Keep existing models bit-identical; no config flags or format changes.	Tests that would have caught it Regression tests that fail on the old behavior, not just pass on the new.

Tech Stack

LLM Serving & Inference

Orchestration & Infrastructure

Data Engineering

ML / AI

Languages

Contributions

Education

B.Tech, Civil Engineering — BIT Sindri, Dhanbad · 2019–2023 · CGPA 8.99