Introducing RustyStats: Fast GLM Engine with Regularisation

Spent the last few days cleaning some really messy import data and wow... real-world data is NO JOKE! 😅 I built a Python script to clean vehicle import records. The data had: - Phone numbers stuck to addresses with no spaces - "CHASIS" spelled 5 different ways - Random text everywhere (S/O, D/O mixed in addresses) - Engine numbers like "4,160CC" vs "4160CC" vs just "4000" Wrote a bunch of regex patterns to extract: ✓ Phone numbers ✓ Clean addresses   ✓ Chassis numbers ✓ Engine specs ✓ Model years Took way more iterations than I expected. Every time I thought I was done, found another edge case 😂 The script now handles most patterns automatically. Not perfect, but getting 85-90% clean data from what was basically unstructured text feels like a win. Main takeaway: Data engineering is less about fancy algorithms and more about dealing with messy reality. And regex. Lots of regex. #DataEngineering #Python #DataCleaning #LearningToCode

🚀 Same Dataset, Different Languages – Concept First! Understanding the concept matters more than memorising a specific language. Once you know what you want to do, you can implement it in SQL, Python, Shell, Excel, or any tool. Goal: Find total approved claims per patient SQL SELECT patient_id, SUM(claim_amount) AS total_claim FROM claims WHERE claim_status='Approved' GROUP BY patient_id; Python import pandas as pd claims = pd.DataFrame({   'patient_id':[101,101,102,103],   'claim_amount':[12000,30000,7000,150000],   'claim_status':['Approved','Approved','Rejected','Approved'] }) approved = claims[claims['claim_status']=='Approved'] total_per_patient = approved.groupby('patient_id')['claim_amount'].sum() print(total_per_patient) Shell Script awk -F, '$4=="Approved" {sum[$2]+=$3} END {for (i in sum) print i,sum[i]}' claims.csv Excel Formulae =SUMIFS($C:C, $B:B, F2, $D:D, "Approved")

#python "Dirty" Input Data Manual data cleaning is time-consuming and prone to human error. If a dataset has 100 columns, checking each one for NaN values or out-of-range numbers individually is inefficient. The Solution: A Validation Pipeline Using the Pandas and Great Expectations (or simple custom logic) libraries, we can build a script that automatically "quarantines" bad data and cleans the rest. import pandas as pd import numpy as np def clean_data_pipeline(df): # 1. Deduplication df = df.drop_duplicates() # 2. Handle Missing Values # Fill numeric columns with the median, categorical with 'Unknown' numeric_cols = df.select_dtypes(include=[np.number]).columns df[numeric_cols] = df[numeric_cols].fillna(df[numeric_cols].median()) categorical_cols = df.select_dtypes(include=['object']).columns df[categorical_cols] = df[categorical_cols].fillna('Unknown') # 3. Data Type Enforcement # Ensure date columns are actually datetime objects if 'timestamp' in df.columns: df['timestamp'] = pd.to_datetime(df['timestamp'], errors='coerce') # 4. Outlier Removal (using Z-Score) for col in numeric_cols: upper_limit = df[col].mean() + 3 * df[col].std() lower_limit = df[col].mean() - 3 * df[col].std() df = df[(df[col] <= upper_limit) & (df[col] >= lower_limit)] return df # Usage raw_data = pd.read_csv("raw_input.csv") cleaned_data = clean_data_pipeline(raw_data) cleaned_data.to_csv("production_ready_data.csv", index=False)

📈 PandasAI: ask your data questions PandasAI is a Python library that lets you analyze and visualize data using LLMs, directly on top of pandas. 🧠 - Ask natural-language questions about your dataset - Spot patterns and anomalies - Compare multiple datasets - Generate, edit, and refine charts on the fly 📂 - CSV - SQL databases - Parquet files No prompt engineering. No context switching. Just “Why is revenue spiking here?” → chart. 🐍 PandasAI doesn’t try to replace pandas — it augments it. Think of it as a conversational layer between you and your data. This is especially useful for: - fast exploratory data analysis - analysts who think in questions, not code - prototyping insights before turning them into production pipelines 🔗 GitHub: https://lnkd.in/dr4ZaxUT 🌐 Live demo / web app: https://lnkd.in/dtKTE2Tr

Quantile regression is a valuable tool for analyzing the relationship between variables, especially when data is not evenly distributed or has outliers. Unlike traditional linear regression, which focuses only on the mean, quantile regression allows us to predict different points across the distribution of the target variable. Challenges: ❌ Compared to linear regression, quantile regression requires more computational power and can be harder to interpret for non-experts. ❌ Larger sample sizes might be needed to achieve stable and reliable quantile estimates, especially for extreme percentiles. ❌ The model's results might be less intuitive if you are accustomed to traditional regression techniques, which could limit ease of communication. Advantages: ✔️ Quantile regression helps to explore trends at various quantiles, offering a more detailed picture of your data. ✔️ This method is highly effective for non-normal data, particularly when there are outliers or heavy tails. ✔️ It is ideal for situations where extreme values or various percentiles are as important as the central trend. How to handle quantile regression in practice: 🔹 R: Use the quantreg package to apply quantile regression. The rq() function allows you to specify the quantiles you're interested in. 🔹 Python: In Python, statsmodels provides quantile regression with the QuantReg() function to analyze different percentiles of your data. The attached visualization is based on a Wikipedia image (link: https://lnkd.in/ejfEbKac) and illustrates quantile regression lines at various percentiles, showing how predicted values differ across the distribution. To explain this topic in further detail, I collaborated with Micha Gengenbach to create a comprehensive tutorial: https://lnkd.in/eDAnBSTM Curious to learn more about statistics and R programming? Join my online course, "Statistical Methods in R." Further details: https://lnkd.in/ed7XyXQm #RStats #Data #Rpackage #database #Statistics

#python Model Decay Imagine you have a model predicting customer churn. If a new competitor enters the market or consumer behavior shifts due to an economic trend, the data your model sees today will look very different from the data it was trained on. The Solution: Automated Drift Detection We can use the evidently or scipy libraries to implement a statistical check (like the Kolmogorov-Smirnov test) that compares the distribution of live data against our training baseline. import pandas as pd from scipy.stats import ks_2samp def check_drift(reference_data, production_data, threshold=0.05): """ Compares two datasets using the Kolmogorov-Smirnov test. Returns columns that have statistically significant shifts. """ drift_detected = [] for column in reference_data.columns: # p-value < 0.05 generally suggests the distributions are different statistic, p_value = ks_2samp(reference_data[column], production_data[column]) if p_value < threshold: drift_detected.append(column) return drift_detected # Example Usage # drifted_cols = check_drift(train_df, current_month_df)

Quantile regression is a valuable tool for analyzing the relationship between variables, especially when data is not evenly distributed or has outliers. Unlike traditional linear regression, which focuses only on the mean, quantile regression allows us to predict different points across the distribution of the target variable. Challenges: ❌ Compared to linear regression, quantile regression requires more computational power and can be harder to interpret for non-experts. ❌ Larger sample sizes might be needed to achieve stable and reliable quantile estimates, especially for extreme percentiles. ❌ The model's results might be less intuitive if you are accustomed to traditional regression techniques, which could limit ease of communication. Advantages: ✔️ Quantile regression helps to explore trends at various quantiles, offering a more detailed picture of your data. ✔️ This method is highly effective for non-normal data, particularly when there are outliers or heavy tails. ✔️ It is ideal for situations where extreme values or various percentiles are as important as the central trend. How to handle quantile regression in practice: 🔹 R: Use the quantreg package to apply quantile regression. The rq() function allows you to specify the quantiles you're interested in. 🔹 Python: In Python, statsmodels provides quantile regression with the QuantReg() function to analyze different percentiles of your data. The attached visualization is based on a Wikipedia image (link: https://lnkd.in/e7eYbpPg) and illustrates quantile regression lines at various percentiles, showing how predicted values differ across the distribution. To explain this topic in further detail, I collaborated with Micha Gengenbach to create a comprehensive tutorial: https://lnkd.in/eyb_DFr8 Curious to learn more about statistics and R programming? Join my online course, "Statistical Methods in R." Take a look here for more details: https://lnkd.in/d-UAgcYf #data #rstats #coding #package

Still talking about data structure having considered list and tuple I am wrapping up the other two which are: Dictionary and Set. While dictionary is a mutable, ordered and efficient collection, Set is unordered collection of unique elements. Set do not allow for duplication. An amazing part of this rundown. # Dictionary are created using curly bracket {} with keys and value. The value pair are separated by commas # key must be immutable data type like strings, number etc #Value can be of any data type which can include list or nested dictionary. I will write some example of syntax for dictionary. Consider these: my_dict ={"name": "jamie" , "age": 30, "city": "Lagos" ,"country" , "Nigeria" } lets play a bit print(my_dict["name"}) # state what will be the output in the comment print(my_dict) # what will the output be? Now, looking at Set. Set is an unordered collection of unique element. They are defined using curly bracket {} but does not have key -value pair. let jump into looking at the examples. When we consider set, we have something like: my_set ={1, 2, 3, 4, 4, 5, } print(my_set) output {1, 2, 3, 4, 4, 5, } my_set .add(6) print(my_set) now output becomes {1, 2, 3, 4, 4, 5, 6} wow both Dictionary and Set are wonderful data structure in python, making data manipulation great. what is your experience with Dictionary and set data structure I will love to hear from you. #python #Datastructures #Dictionary #set