Avijit Swain

🚀 Building a Time Series Forecasting Project - EDA Analysis Complete   Continuing the structured time series analysis and forecasting project using real market data (MSFT stock prices).   After completing Notebook 01 (Data Inspection), this notebook dives into deeper time series diagnostics to understand the statistical structure before moving into modeling.   Previous Post: https://lnkd.in/dxgEatzB   This step establishes the statistical groundwork for feature engineering and model development.   𝐖𝐡𝐚𝐭 𝐭𝐡𝐢𝐬 𝐩𝐡𝐚𝐬𝐞 𝐟𝐨𝐜𝐮𝐬𝐞𝐝 𝐨𝐧:        • Horizon selection based on market and data regimes        • Multi-horizon trend analysis (weekly/monthly smoothing)        • Decomposition (classic & STL) with critical interpretation        • Stationarity diagnostics of log returns        • ACF/PACF analysis of log-returns to assess linear memory        • Volatility clustering and rolling risk behaviour        • Calendar effect testing in mean and variance   𝐊𝐞𝐲 𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲𝐬:        • Price levels are non-stationary and trend-dominated.        • Log-returns provide a more stable modeling foundation.        • Volatility is time-varying (heteroskedastic), not constant.        • Log-returns show weak linear autocorrelation — suggesting limited short-term memory.        • No statistically significant calendar effect in mean returns   𝐍𝐞𝐱𝐭 𝐮𝐩 → Notebook 03 intent to construct feature representations that explicitly capture temporal dependence and volatility structure for downstream modeling.   Tools: Python | Pandas | NumPy | Matplotlib | Seaborn | Statsmodels | SciPy | yFinance   Full repository: https://lnkd.in/d7idKJVT   Notebook attached as PDF for reference.   + Follow Shashank Garewal for the more such data science content and insights.   #TimeSeries #DataScience #Python #Forecasting #QuantFinance #MachineLearning #Analytics #StockForecasting #Microsoft

7

1 Comment

🔹 Fractal – Databricks Interview Experience (Data Engineering Round) ------------------------------- ❌ Most candidates fail Fractal’s Databricks round — not because they can’t write code, but because they don’t understand the platform deeply enough. Fractal’s Databricks interview is all about platform mastery: clusters, jobs, Delta, governance, performance, orchestration, and real production debugging — not PySpark alone. Here are fresh, never-used-before, Databricks-only scenario questions: 2025 1️⃣ Your Databricks Job uses a job cluster with autoscaling. For the first 5 minutes, executors keep scaling up and down rapidly. What Databricks-specific misconfiguration could cause “autoscaling thrashing”? 2️⃣ A Databricks notebook runs successfully when executed manually, but fails only when triggered as a Databricks Job. What workspace-level and job-cluster-level differences will you investigate? 3️⃣ Your Delta table is ACID compliant, but you notice that DELETE operations are taking unusually long in Databricks. What internal Delta Lake + Databricks mechanisms cause slow delete performance? 4️⃣ Databricks Repos merge conflict resolution still shows old notebook versions even after resolving the conflict in Git. What Databricks-specific caching or sync behavior explains this? 5️⃣ A reviewer in Fractal asks: “How do you prove your Databricks job has end-to-end data lineage?” What tools and workspace settings in Databricks provide lineage beyond Unity Catalog? 6️⃣ A Photon cluster fails with the error: Operation requires native execution but fallback is disabled. What Databricks setting or code path can trigger this? 7️⃣ Your Databricks streaming job slows down over days even though micro-batch sizes remain stable. What Databricks workspace-level causes can lead to state store bloat? 8️⃣ A Delta Live Table (DLT) pipeline stops with the message: “Event log exceeded maximum retention limit”. What DLT-specific configuration is responsible? 9️⃣ A SQL Warehouse dashboard refresh takes 8 minutes but the underlying SQL query runs in 15 seconds. What Databricks Warehouse features can cause UI-level render delays? 🔟 Your cluster library installation fails only during job execution but succeeds interactively. What Databricks job isolation behaviors cause library install differences? 1️⃣1️⃣ You enabled Delta Universal Format (UniForm) in a table. Suddenly your S3/GCS storage costs increase. Which Databricks UniForm behavior leads to higher storage usage? 1️⃣2️⃣ Your Unity Catalog external location validation fails even though cloud permissions are correct. What workspace-level settings can still cause UC validation to break? --------------- 💬 fractal focuses on real production reliability — not just PySpark syntax. They test your ability to design resilient, compliant, and automated Lakehouse pipelines. 📞 DM or WhatsApp +91 93594 45862 to access: ✅ Real Databricks Question Bank ✅ Mock Interviews ✅ End-to-End Project Practice ✅ Pay After Placement Options

2

🎬 Sentiment Analysis: Then vs. Now 👉 Input: “The movie was not bad at all.” 👉 Goal: Predict Sentiment → Positive 🔹 How Traditional NLP Worked 1️⃣ Feature Extraction → Used Bag of Words / TF-IDF. ⚠️ Issue: Words treated independently. “bad” = negative, even in “not bad.” 2️⃣ Machine Learning Models → Logistic Regression, SVM on handcrafted features. ✅ Simple, but only as smart as the features we designed. 3️⃣ Deep Learning (RNNs, LSTMs) → Added sequential context. ⚠️ Limitation: Struggled with long sentences, sarcasm, nuance. Result: Often failed to capture meaning in real context. 🔹 How LLMs Changed the Game 1️⃣ Tokenization → Break text into subwords (e.g., “unbelievable” → “un”, “believe”, “able”). 2️⃣ Embeddings → Map tokens into vectors that capture semantic relationships. 3️⃣ Transformers → Process the entire sequence in parallel using self-attention. 4️⃣ Attention Mechanism → Learns which words matter. ✔️ Example: understands “not” flips the meaning of “bad.” 5️⃣ Scaling Up → Trained on massive datasets with billions of parameters. LLMs (GPT, LLaMA, PaLM) become general-purpose foundation models. ✅ Result: LLM correctly predicts sentiment = Positive because it captures context + meaning. 📌 Takeaway Traditional NLP = brittle, task-specific. LLMs = contextual, scalable, general-purpose. They’re now the backbone of summarization, translation, coding, Q&A, and more. 💡 Terms like tokenization, embeddings, attention, transformers, fine-tuning, zero/few-shot will be unpacked in upcoming posts. Stay tuned! Follow Udit Soni for more! #GenAI #LLM #NLP #ArtificialIntelligence

14

COVID-19 brought abrupt changes in consumer behavior and impacted the accuracy of forecasting models. Similarly, UPI (Unified Payment Interface) abrupted consumer's spending methods, bringing many of them online. Concept drift happens when the statistical properties of the target variable, which the model is trying to predict, itself change over time. This causes problems because the predictions become less accurate and become unreliable. Here's a short video for Data Science enthusiasts I have created on the topic of Data drift and model monitoring followed by a practical exercise at the end. Like and subscribe for more such interesting concepts. Also, like and share over here for maximum reach. : ) Video Link: youtu.be/tQjRQWfYQ10 YT channel Link:  youtube.com/@datatrek #datatrek #datascience #machinelearning #statistics #deeplearning #ai

3

Sharing my handwritten notes on Functional Forms in Regression Models ✍️📒 Choosing the right functional form is critical in econometrics – it can completely change how we interpret relationships between variables. In these notes, I’ve explained: * Why functional form matters * Linear vs log-linear vs double-log forms * Semi-log models and their interpretation * How to test and select the right form * Worked-out examples for clarity I’m attaching the notes below for easy access. These should help you quickly revise and understand how functional forms shape economic analysis. Let me know if you’d like me to continue sharing handwritten notes on other econometrics topics. Join the Arthapoint community here https://lnkd.in/g3Q3D96h

73

Over the years working with large datasets, I’ve realized something simple: 👉 Most performance issues don’t start with queries, they start with design. That’s why I’m starting a new series on SQL Optimization & Scalability — sharing practical lessons on schema design, indexing, partitioning, and more. Here’s 💡 Part 1: Faster SQL Starts with Smarter Schema Design & Indexing. While working in today’s evolved data industry, I’ve realized one thing very clearly: 👉 A well-designed schema solves 90–95% of business problems before we even start writing complex queries. Why? Because a robust schema and efficient database design can make or break how quickly we can turn raw data into insights — whether for day-to-day reporting, powering ML models, or feeding dashboards. At first, schema design feels simple when you’re only handling a few thousand rows. But as data grows exponentially, Data Definition Language (DDL) decisions suddenly become critical. The right schema design = faster query retrieval. Efficient DDL = smoother Data Manipulation Language (DML) operations at scale. Together, they set the foundation for every downstream use case — from analytics to AI. This series will explore how schema design, indexing, partitioning, and other optimizations unlock query speed and scalability. 💡 My takeaway so far: Schema is not just a database step — it’s a business impact multiplier. Indexing: The Shortcut That Makes SQL Fly When dealing with millions of rows, scanning data sequentially (O(N)) quickly becomes a bottleneck. That’s where indexing comes in — reducing lookups to O(logN) by letting SQL jump directly to the relevant data. Indexes build sorted, searchable structures (like B-Trees, Bitmaps, Columnstores), acting as smart catalogs that avoid expensive full table scans. 🔹 Clustered Index Defines the physical row order in a sorted manner. 👉 Think of it like a diary arranged strictly by date. If you want to find March entries, you flip straight to March — without reading Jan and Feb. CREATE CLUSTERED INDEX cix_fact_transact_date on dbo.fact_transactions(transaction_date). And QUERY SELECT * from dbo.fact_transactions WHERE transaction_date between BETWEEN '2024-01-01' AND '2024-01-31';   🔹 Non-Clustered Index Creates shortcuts for specific query patterns (great for filters like city + date). 👉 It works like a table of contents — pointing directly to the page you need. CREATE NON-CLUSTERED INDEX nix_fact_city_date on dbo. fact_transactions(city_id,transaction_date) INCLUDE (price); Query SELECT transaction_date, price FROM dbo.fact_transactions WHERE city_id = 1001 AND transaction_date >= '2024-01-01' 💡 Takeaway: 🔹 Indexes aren’t just “performance tweaks.” They’re query accelerators that transform how databases scale. 🔹 Use clustered indexes for natural order. 🔹 Use non-clustered indexes for targeted lookups. #SQL #QueryOptimization #Database #DataScience #DataEngineering

8

🎓 Day 11: Mastering Table Calculations in Tableau 🧠 What Are Table Calculations? Table Calculations are computations applied after the data has been aggregated and displayed in the view. They operate on the result of the visualization, not the raw data. 📌 Examples: 1. % of total sales 2. Running totals 3. Year-over-year growth 4. Rank of products within category 5. Moving averages ⚙️ Why Use Table Calculations? Show % of total per sub-category, Percent of Total visit www.siradeel.blogspot.com for full details about these sections 🛠️ Basic Table Calculations (No Formula Required) Tableau gives you built-in table calcs through right-click: Try This: Open a new sheet Drag Sub-Category to Rows Drag Sales to Columns Now: Right-click on SUM(Sales) in the view Choose Quick Table Calculation Select: ✅ Percent of Total ✅ Running Total ✅ Rank 🔍 You’ll see how numbers change instantly — that’s a table calc! 📐 Advanced Table Calculations (Using Functions) Let’s create our own calculation now. ✍️ Example 1: Calculate Percent of Total Manually Create a new calculated field: tableau CopyEdit SUM([Sales]) / TOTAL(SUM([Sales])) This divides each row’s sales by the total across the table. Tableau will ask: “Compute using?” ➤ Choose the direction (Table Across, Down, Pane, etc.) 🎯 Understanding Compute Using visit www.siradeel.blogspot.com for full details about these sections 🎓 Tip: Always check "Edit Table Calculation" to set this properly! 🧪 Common Table Calc Functions visit www.siradeel.blogspot.com for full details about these sections 🧪 Hands-on Example: Year-over-Year Growth (YOY) Goal: Compare this year’s sales to last year’s Steps: Drag Order Date to Columns → Choose Year Drag Sales to Rows Create a new calculated field: (SUM([Sales]) - LOOKUP(SUM([Sales]), -1)) / LOOKUP(SUM([Sales]), -1) Format it as percentage 🔍 This shows YOY growth using LOOKUP() to reference the previous year! 🧪 Challenge Task (Try Now!) Goal: Show top 5 sub-categories by running sales total Create a bar chart: Sub-Category vs Sales Sort it in descending order Create a new field: RUNNING_SUM(SUM([Sales])) Add a filter to only show top 5 using INDEX() <= 5 This teaches you to combine ranking + window calculations. 🔄 Table Calculation Best Practices ✅ Use "Edit Table Calculation" to control computation direction ✅ Use tooltips to explain what each number means ✅ Combine with filters for interactive dashboards ✅ Be cautious when changing table structure — table calcs can break

3

1 Comment

‎🌟 Day 7 — The Process Matters ‎ ‎I love CRISP-DM because of its smooth flow in data analysis. ‎It involves six steps: ‎1️⃣ Business Understanding ‎2️⃣ Data Understanding ‎3️⃣ Data Preparation ‎4️⃣ Modeling ‎5️⃣ Evaluation ‎6️⃣ Deployment ‎ ‎Most analysts stop at data preparation, bringing insights that guide decision-making — but that’s just the beginning for data scientists! 🚀 ‎ ‎📖 “The end of a matter is better than its beginning.” (Ecclesiastes 7:8) ‎ ‎Every stage counts — both in data and in life. 💪 ‎How true is this? 🤔 ‎ ‎#15DaysChallengeWithDataLiving #15DaysOfConsistencyWithDataLiving #CRISPDM #DataAnalytics ‎ ‎

7

Sarvam-M: A 24B Parameter Multilingual AI Model is finetuned Mistral-Small with Indic language desi nuances covering 11 major languages from india 🇮🇳 Key Features: • Dual-mode interface: Quick "non-think" responses + detailed "think" mode for complex reasoning 🧠 • Strong performance on math (GSM-8K) and coding (SWE-Bench) benchmarks 📊 • Supports both native Indic scripts and romanized text ✍️ • Built for real-world multilingual conversational agents 💬 I think it's a good start , instead of training model from scratch Sarvam focused on building indic dataset and model like in video shows good translingual conversational capability . Try it out via the Colab link - https://lnkd.in/gzcW4hru Playground - https://lnkd.in/gXijZrkY

27

1 Comment

📊 Turning Data into Growth Stories | Sales Performance Dashboard 🚀 This snapshot proves why analytics is the backbone of smart business decisions 👇 🔹 $2.33M Total Sales with strong YoY growth 🔹 $292.3K Profit → profitability moving in the right direction 🔹 Returns under control at just 5.81% 🔹 Consistent upward sales trend vs previous year 🔹 Top 5 states driving the majority of revenue 🔹 Technology & Office Supplies leading category performance 📈 What stands out? It’s not just higher sales—it’s better margins, controlled returns, and regional focus. 💡 Key takeaway for leaders & analysts: ✔ Track YoY performance, not just totals ✔ Focus on profitable categories, not only volume ✔ Optimize regions that consistently outperform ✔ Use data to reduce returns and increase trust Data doesn’t just report the past — it shapes the future. If you’re passionate about data-driven growth, this is your kind of dashboard. #DataAnalytics #BusinessIntelligence #SalesAnalytics #PowerBI #DashboardDesign #DataDriven #GrowthMindset #AnalyticsTrends #LinkedInAnalytics

32

Polynomial Regression helps us understand non-linear relationships where straight lines fail 📈 By transforming features into polynomial terms, we can model real-world trends more accurately. Perfect for salary prediction, demand forecasting, and growth analysis 🚀 Save this post to revise Machine Learning basics! #PolynomialRegression #MachineLearning #DataScience #SupervisedLearning #RegressionAnalysis #MLBasics #AI #PythonForDataScience #DataAnalytics #LearnMachineLearning #Statistics #DataScienceStudent

1

Just came across this gem again — Forecasting: Principles and Practice, The Pythonic Way. https://otexts.com/fpppy/ It was first introduced to us by Prof. Anant Agarwal during my Postgraduation for the Time Series Analysis subject. Now that it's available in Python too, it's a powerful resource for anyone working on forecasting and time series analysis. Highly recommend for data folks! 🔍📈 #TimeSeries #Forecasting #Python #DataScience

21

1 Comment

Column-based vs Row-based file formats — a simple example While learning about column-based file formats like Parquet, I finally understood why they are so efficient for analytics. Sharing the understanding with a very simple example 👇 Imagine this data: Name, City, Salary A, BLR, 50K B, BLR, 60K C, BLR, 55K Now the task is to calculate average salary. Row-based format (CSV): Data is stored like this: A, BLR, 50K B, BLR, 60K C, BLR, 55K Even though we need only Salary, the system still reads: Name City Salary ❌ Extra data read ❌ Slower analytics Column-based format (Parquet): Data is stored like this: Name → A, B, C City → BLR, BLR, BLR Salary → 50K, 60K, 55K Now, to calculate average salary: Only the Salary column is read ✅ Less data read ✅ Faster processing ✅ Better performance Bonus insight: Repeated values (like BLR) are compressed much better in column-based formats, reducing storage and improving speed. Key takeaway: Row-based formats → Best for transactions Column-based formats → Best for analytics & reporting This clarity really helped me today—sharing in case it helps someone else too 😊 Codebasics Dhaval Patel Hemanand Vadivel NAVEEN S #DataAnalytics #Parquet #DataEngineering

9

📊 Unlocking Insights with Tableau: Turning Raw Data into Smart Decisions In the era of big data, making sense of information is crucial—and Tableau makes that easy. At Natlov Technologies Pvt Ltd, we empower teams to turn complex datasets into clear, actionable insights. 🔍 Why Tableau? ✅ Drag-and-drop dashboards ✅ Real-time data visualization ✅ Connects with Excel, SQL, Redshift, Snowflake & more ✅ Interactive, shareable reports ✅ Drives faster, smarter business decisions 💡 Whether it’s customer behavior, sales trends, or operations—Tableau helps decision-makers see the story in their data. 🚀 Pro Tip: Supercharge Tableau with Delta Lake, Redshift, or Azure Data Factory to build powerful BI pipelines. 📣 Are you using Tableau to power your insights? Share your favorite use case below! 👇 #Tableau #DataVisualization #Analytics #BusinessIntelligence #NatlovTechnologies #DashboardDesign #DataDriven #BI #DataAnalytics #SmartDecisions

1

https://lnkd.in/gY7v8Sjj From Data to Decisions: A Story-Driven Sales Analysis with Tableau Public Today, I’m excited to share a Tableau Public Sales Story I created — a narrative-driven analysis that turns raw sales data into actionable insights. In today’s business world, a chart alone isn’t enough — context and narrative are key to making data meaningful. That’s where Tableau Stories shine: they combine interactive visualizations with captions to guide your audience through your thought process — just like a presentation, but richer and more interactive. Here’s what I focused on: 🔹 Understanding Overall Sales Performance: The opening view provides high-level insight into total sales across categories — a foundation for deeper analysis. 🔹 Analyzing Trends Over Time: By exploring sales and profit trends over multiple periods, we uncover patterns that may relate to seasonality or strategy changes. 🔹 Breaking It Down by Category & Region: Segmenting sales by product categories and regions helps reveal which areas are driving growth and which need attention. 🔹 Highlighting Key Opportunities & Challenges: Through the story’s progression, we surface where profit margins are narrowing despite strong sales and where specific sub-segments may be underperforming. 💡 What makes a story different from a dashboard? It’s not just “what the numbers are” — it’s how they evolve, compare, and connect. Each story point builds on the previous one to enhance understanding. #DataVisualization #Tableau #Analytics #BusinessIntelligence #DataStorytelling

20

3 Comments

🚀 Day 24 – Weaviate RAG over PDFs 🚀 Today, I explored building a Retrieval-Augmented Generation (RAG) pipeline using Weaviate and LangChain! Key highlights: Loaded and chunked PDF documents for semantic search. Used OpenAI embeddings (text-embedding-3-small) to create vectors for each document chunk. Stored and queried vectors in Weaviate, enabling fast, scalable retrieval. Built a RAG chain with ChatOpenAI (gpt-3.5-turbo), allowing context-aware answers from documents. Tested queries like summarizing documents and extracting key facts. This approach makes it easy to turn unstructured PDFs into a knowledge base you can query naturally. 💻 GitHub repo with code: https://lnkd.in/guSRDkFM #GenerativeAI #LangChain #Weaviate #RAG #OpenAI #AI #NLP #Python #MachineLearning #KnowledgeManagement

24