Blog
Notes on evaluating AI systems
Practical writing on eval datasets, LLM judges, agent evaluation, and quality gates, with real numbers wherever possible.
Oct 10, 2026 · 2 min read
How Many Evals Are Enough?
Your eval score went from 84% to 86%. Is that a real improvement, or noise? A practical guide to sizing eval sets.
Oct 3, 2026 · 2 min read
Your LLM Judge Is a Measuring Instrument. Calibrate It.
LLM-as-judge is only as trustworthy as its agreement with humans. How to measure that agreement, and what to do when it's low.
Jun 11, 2026 · 1 min read
What if your software could explain itself? AI agent mapping user journey to a software architecture
A video walkthrough of an AI agent that maps a user journey onto the software architecture behind it.
Jun 4, 2026 · 7 min read
The Only Three Levers That Actually Reduce LLM Costs
When teams start building AI-powered products, the first optimization question is usually: “Should we use a cheaper model?” That’s a good question, but it’s only one piece of a much bigger…
Mar 17, 2026 · 3 min read
Understanding RAG Retrieval Strategies: How to Get the Best Answers from LLMs
Modern AI applications increasingly rely on Retrieval-Augmented Generation (RAG) to answer complex queries. RAG combines retrieval from a knowledge base with LLM generation for more accurate,…
Mar 11, 2026 · 2 min read
Building a Kill-Switch for Your AI Agent: Stopping Execution Safely
In the world of AI agents and autonomous workflows, it’s easy to imagine the “rogue AI” scenario: your agent starts reasoning, executing tasks, or modifying state, and suddenly you realize — you…
Mar 7, 2026 · 9 min read
An AI Agent That Investigates Broken Builds
AI coding tools have made writing code faster than ever. Cursor, Copilot, ChatGPT — you describe what you want, the code appears, you ship it. Velocity is up. PR volume is up. And CI failures?…
Feb 27, 2026 · 5 min read
When Your Safety Net Catches Your AI: Why Validation Layers Matter in LLM Applications
I was running my AI-powered release risk agent against a set of Eval test cases when something interesting happened. The system threw an error — and it was exactly the right thing to do. Let me…
Feb 19, 2026 · 6 min read
I Got Interviewed by an AI — and It Asked Me the Same Question Twice
Recently I had an interview conducted by an AI system. It was actually pretty interesting — and a little funny. At one point the AI asked me: “If you work onsite, which state will you work from?”…
Feb 15, 2026 · 8 min read
Extending Promptfoo with RAGAS: Adding RAG-Specific Metrics to Your Eval Pipeline
Promptfoo is excellent at what it does: comparing prompts, running assertions, and catching regressions in CI. But if you’re building a RAG pipeline, Promptfoo’s built-in assertions — contains,…
Feb 9, 2026 · 3 min read
LLM-as-Judge Pattern
Using one LLM to evaluate another LLM’s output is one of the most powerful eval techniques. It is also one of the most dangerous if done carelessly. How to Write a Judge Prompt A good judge prompt…
Feb 5, 2026 · 6 min read
Exploring Dialogflow CX: How Google Thinks About Conversational AI at Scale
I’ve been experimenting with Dialogflow CX, Google’s conversational AI platform, and it feels meaningfully different from what we usually think of as a “conversational agent.” That might be…
Feb 1, 2026 · 9 min read
Single-Turn vs Task-Based (Multi-Turn) Evaluation
Most AI agent failures don’t come from “bad prompts.”They come from treating agents like static functions instead of dynamic systems. Once an agent can: …it is no longer a prompt.It is a…
Jan 16, 2026 · 5 min read
Putting It All Together: A Practical Guide to LLM / AI Evaluation Methods
When I first started learning about LLM and AI evaluations, I struggled to put all the different evaluation methods into a single mental model. Every blog, paper, or tool seemed to focus on one…
Jan 12, 2026 · 4 min read
Your LLM Benchmark Is Lying to You
Everyone wants to know which LLM is “better at coding.” We point to leaderboards, single scores, colorful charts, and declare a winner. But if you’ve ever shipped software to production, you…
Jan 6, 2026 · 1 min read
Even iguana can be 10x developer
Last weekend I spent time setting up an “advanced agents” coding workflow — the kind that promises to magically build applications for you while you sleep. Well, the only thing that didn’t sleep…
Dec 31, 2025 · 1 min read
Build ATS-Optimized Resumes with AI | Full Feature Tour Job Search
I have built AI-powered web application for managing resume alongside job postings and applications. Features intelligent resume optimization, job description analysis, and interview preparation…
Dec 23, 2025 · 1 min read
How I Built an AI Content Generator with LangChain + OpenAI (and a Raccoon)
In this video, I’ll walk you through how I use LangChain and OpenAI models to automatically generate engaging content and hashtags for my social media app — featuring our star raccoon 🦝. You’ll…
Dec 11, 2025 · 1 min read
How Cursor AI and Sentry MCP Turns You Into a 10x Debugger
Unlock a next-level debugging workflow by combining Cursor AI with Sentry MCP. In this video, I walk through a real example of how AI can detect, analyze, and fix application issues with almost no…
Nov 15, 2025 · 2 min read
Level up your AI Engineering skills
Not that long ago, I set a personal goal — to dive deep into AI and see how far it can actually go in making life and work easier. Call me lazy, but I’m all about working smarter, not harder. So,…
Nov 3, 2025 · 2 min read
Model Context Protocol (MCP) for AI-Powered Browser Automation
Model Context Protocol (MCP) is great for AI-powered browser automation and debugging, with notable implementations from Playwright MCP by Microsoft and Chrome DevTools MCP by Google. These two…
May 27, 2025 · 3 min read
The Evolving Role of SDETs: From Testers to Quality Engineers
Software testing isn’t dying—it’s evolving. In many industries outside of Finance, Healthcare, or Defense, there’s a growing tolerance for minor bug escapes. Not because quality is being ignored,…
Feb 21, 2025 · 1 min read
Bug Reporting Made Easy with Chrome DevTools AI Assistant
Bug reporting is a critical part of software development, but it can often be time-consuming and require deep debugging skills. Chrome DevTools AI Assistant aims to simplify this process by…
Feb 10, 2025 · 3 min read
Monorepo pre-commit hooks with Husky
Managing a monorepo brings incredible benefits, such as code reuse and unified project management, but it also comes with unique challenges, particularly when it comes to test automation. Running…
Feb 7, 2025 · 2 min read
AI vs SDET or SDET and AI ?
There’s a lot of talk about AI replacing SQE and SDETs. Here’s why AI falls short when it comes to testing: 1. AI Can’t Architect Test Automation Frameworks AI can generate test cases, but who…
Jan 27, 2025 · 2 min read
Monitoring and debugging your app with Sentry.io
As a test engineer, observability tools are a crucial part of my workflow. They help me triage errors, understand user behavior, and map out service dependencies. This weekend, I decided to dive…
Jan 21, 2025 · 3 min read
How to make sure the Code and Libraries are Secure with SonarQube
Last year, I was interviewing at a cybersecurity company, and one of the questions they asked was: How do you make sure the code and libraries that you use are secure? I knew about security…
Nov 2, 2024 · 2 min read
How to add code formating to your JavaScript project
To set up linters and code formatting for a React and Express project, you’ll want to use ESLint for linting and Prettier for consistent formatting. Here’s a step-by-step guide for frontend and…
Oct 31, 2024 · 2 min read
How to add Swagger OpenAPI docs to NodeJS Express
Setting up OpenAPI (Swagger) in your Express app allows you to generate interactive API documentation. Follow these steps to integrate OpenAPI with your Express app. 1. Install Dependencies You’ll…
Oct 30, 2024 · 3 min read
Docker setup for NodeJS application
Here’s a Docker setup to containerize both the React frontend and the Express backend efficiently. Below are the two separate Dockerfiles and a docker-compose.yml to orchestrate them. 1.…
Oct 28, 2024 · 3 min read
How Playwright Communicates With Browsers
Playwright communicates with browsers through browser-specific protocols over a WebSocket or process communication channel. Each supported browser (like Chromium, WebKit, and Firefox) exposes an…
Oct 8, 2024 · 5 min read
A Comprehensive Guide to Initiating a Test Automation
Many companies I’ve worked for or interviewed with have faced a common challenge: successfully initiating a test automation project. During interviews, it’s common for interviewers to mention that…
Jul 20, 2024 · 6 min read
E2E Test Automation Design Patterns
Throughout my career, I have worked with various test automation frameworks, including Selenium WebDriver (Java, Python), WebdriverIO (JavaScript), Puppeteer (JavaScript), Cypress, and Playwright…
Jul 2, 2024 · 3 min read
iOS Mobile UI Testing with XCUI Test
I recently worked on an iOS ToDo list application for a couple of reasons: to learn more about SwiftUI and to have a pet project for practicing XCUI Test. This simple ToDo list application uses…
Jun 16, 2024 · 1 min read
Postman Generative AI Backend API Testing
In this tutorial, I’m exploring Postman Generative AI for Backend API Integration Testing
May 21, 2024 · 10 min read
Changing Quality Culture, from Startup to Enterprise (shift left)
I recently presented at the Chicago QA Association, where I talked about changing the quality culture in the organization. As a company progresses through various stages, it’s crucial to ensure…
Mar 7, 2024 · 1 min read
Using Postman proxy for data setup
Quick video on how to use Postman Proxy for website data setup automation
Dec 27, 2023 · 2 min read
E2E Testing Framework Design
When building an E2E testing solution it is important to understand how the application works not only from UI (black box) perspective but also how backend services interact one with another, how…
Oct 12, 2023 · 4 min read
Cypress vs Playwright (2023)
I’m always looking for ways to improve testing and team performance, as described in the book “Accelerate” having fast feedback is crucial for high-performing teams. So lately, I’ve come across…
Sep 5, 2023 · 4 min read
Improving Software Testing Efficiency: Why E2E Tests Are Slow and How to Speed Them Up
In the fast-paced world of software development, speed is paramount. Being the first to market can make all the difference, allowing you to reap the rewards of your hard work. Software testing is…
Jun 20, 2023 · 6 min read
Fixing Flaky E2E Tests in Cypress
One of the biggest challenges in E2E testing is to overcome flaky tests. In this blog post, I will share a few approaches that I use to reduce flakiness in Cypress (the test framework I’m using…
May 21, 2023 · 4 min read
Understand E2E Test Automation
Almost everywhere I worked management (including QA managers) and developers have different perspectives on the role of test automation and how it works. In this post, I will try to explain my…
Sep 7, 2022 · 3 min read
Re-inventing testing pyramid
The testing pyramid introduced by Mike Cohn in his book Succeeding with Agile (2009), originally had only 3 tiers, given it was more than a decade ago we should ask ourselves is that still…
Nov 25, 2020 · 2 min read
Python API testing beyond the GET request.
Recently I wrote a blog post on API testing in Python, but I only talked about GET requests there. In this tutorial we are going to look at POST and PUT requests and how to provide data for it.…
Nov 15, 2020 · 3 min read
Performance testing with JMeter
JMeter is a powerful tool for load and performance testing. In this tutorial we going to look on how to install JMeter on MacOS, record and run your test. To install JMeter on Mac open up the…
Nov 14, 2020 · 2 min read
Visual Testing with Jest and Puppeteer
In this tutorial we are going to create a Visual Regression Tests with Jest and Puppeteer. If you not familiar with visual regression testing, it is tests based on image recognition, basically we…
Nov 11, 2020 · 1 min read
Django Database Model Unit Testing
Working with databases always have some challenges. The main problem is if we run tests against real database it can get flooded with fake data and yes you going to run apps against test…
Nov 11, 2020 · 2 min read
Health Check of a Public Website using Python
So you running public website and you have something like Jenkins server where you can execute small programs and monitor state of your website. There are many advanced paid solutions for that,…
Nov 4, 2020 · 2 min read
Detox Test Reports in Jenkins
In this blog post we going to integrate Detox reports in Jenkins CI/CD pipeline. But first let’s see how test reports working in Jenkins. Most likely you going to have JUnit plugin pre-installed…
Oct 29, 2020 · 3 min read
Debugging Detox UI tests in React Native
Those who worked with mobile applications UI testing in Appium used to elements finder which comes with Appium Inspector, but when you work with Detox finding elements is quite challenging. In…
Sep 24, 2020 · 2 min read
API testing with Python
There are many ways of doing API testing, the most popular tools is Postman (JavaScript) and RestAssured (Java), in this post we going to use Python request library to test Deck of Cards API.…
Sep 4, 2020 · 2 min read
Automating Checkers with Selenium in Python
I’m originally from Russia and since I was a kid I been playing Chess and Checkers. I have heard about competitions between humans and computers, and it was always interesting to me what is going…
Aug 23, 2020 · 1 min read
Configure NodeJS in Jenkins
I’m currently working on ReactJS and Reactive Native application in order for me to build the app in Jenkins I need to install NodeJS plugin and configure it. Here is how to do it. Assuming you…
Aug 7, 2020 · 1 min read
JavaScript UI Testing with Puppeteer: Improving Click function
In ReactJs project I’m working on we are using Puppeteer as UI testing framework, the framework is part of node dev dependencies and living with app source code. Since development team constantly…
Jul 30, 2020 · 1 min read
Changing iOS and Android simulator status bar for Visual Testing
There is lots of talk lately about visual testing and benefits of catching UI bugs upfront and even though it’s pretty easy task for modern web browser testing, it can be complicated for mobile…
Jul 22, 2020 · 1 min read
How to access Digital Ocean droplet on Mac / Linux
This is a short how-to guide how to access Digital Ocean droplet from your machine First let’s create a droplet. Assuming you already have digital ocean account, click Create and then Droplets…
Jun 30, 2020 · 1 min read
Using Loading Animation in your Test Automation
In test automation, there is quite often the situation when we need to wait for the page to load before interacting with UI elements on the page. If the page is too heavy and developers don’t want…
Jun 27, 2020 · 1 min read
Retry failed Detox tests with Mocha
UI tests are flaky, especially when you run it on mobile devices. There are multiple factors that can lead to false-positive test results, if that happens you would want retry failed test before…
Jun 25, 2020 · 3 min read
Running JavaScript Appium on Azure DevOps
So you built your Appium tests and running it locally, then developer make changes and asking you to run tests, you dropping everything making a build against his branch for android and ios…
Jun 22, 2020 · 1 min read
React-app with Puppeteer, Jest and Travis CI
E2E testing of React applications could be done with multiple frameworks, including WebdriverIO, Cypress, or Puppeteer. In this blog post, we going to see how to setup Puppeteer with Jest and run…
Jun 13, 2020 · 2 min read
Abstracting Appium Capabilities from test file
In the previous blog we set Appium to run with Jest and added desired capabilities to beforeAll hook. While this approach will work for a small project when you dealing with a fairly large…
Jun 11, 2020 · 2 min read
Mobile Testing with Appium, Jest and WebdriverIO
While there are lots of blogs on how to run Appium with Mocha, there is not much info on how to do it with Jest as a test runner. In this blog post, we will set up Appium with Jest and WebdriverIO…
Jun 9, 2020 · 2 min read
E2E Testing React Native Applications
There are few frameworks that are suitable for React Native apps testing, the most popular is Appium and Detox. Appium is a standalone testing application and Detox is more of a grey box testing…
Jun 4, 2020 · 1 min read
Launch Android emulator on macOS using command line
Testing mobile applications I quite often have to launch Android emulator, there is an option to launch through Android Studio and then UI of AVD. Working in automation space I don’t want all…
Jun 3, 2020 · 1 min read
Xpath cheatsheet
Often I find myself using certain XPath patterns and then forgetting how to do it, so I decided to create XPath cheatsheet. 1. XPath to search by partial id, so let say you have div id which…
May 23, 2020 · 2 min read
How to set environment variables in Node.js
Lately, I been working a lot with JavaScript and Node.js, one thing I struggled is setting up environment variables so I could run my automation tests based on the flag I passed. As a test runner…
Apr 3, 2020 · 2 min read
JavaScript Event Delegation
For my study project, I’m building a Chat application, similar to Slack. And if you ever used Slack you know there are different channels and if there is no channel that you need for the project,…
Mar 23, 2020 · 1 min read
Cypress.io quick reference
This blog post will be used for Cypress.io quick reference, use the official website for more detailed information on how things work in Cypress Starting cypress locally (more detailed info on the…
Mar 18, 2020 · 1 min read
Flask quick reference
This post will be updating constantly to keep different flask commands cheat sheet. Set the environment variable FLASK_APP to be application.py Change Flask running port
Mar 5, 2020 · 1 min read
Cypress.io interacting with tables and lists
Often we get to a situation when we have a list, table or even div elements which have no way to differentiate one from another, here is an example: If you using WebDriver most likely you would…
Mar 1, 2020 · 5 min read
Is working from home worth it?
“People already work remote, they just don’t realize it. Those who work on the first floor, almost never talk face to face with those who work on the second floor” – someone posted it on Linkedin,…
Feb 24, 2020 · 1 min read
Get checkbox value in Flask and Jinja2
Here is how to get checkbox value from jinja2 template in a flask, first let’s create our jinja2 file: Then lets retrieve value from the form: Where rating is the name of input boxes
Feb 23, 2020 · 1 min read
How to use sessions in Flask
One of the things you need to do as a developer of the application is track who is logged in to your application so you can display pages related to that specific user. In Flask you can use…
Feb 22, 2020 · 1 min read
Ignore CSV header in Python
Sometimes you would get a data sheet as CSV file which needs to import in SQL, the problem is CSV files don’t really care what data type in the sheet, unlike SQL. Let’s say you have a books CSV…
Feb 19, 2020 · 1 min read
Pipenv how to create python virtual environment
First things first we need to install pipenv, so make sure you have python 3.* and pip installed on your machine. Then create directory for your project (or navigate to existing one): Once you…
Jan 18, 2020 · 4 min read
South Florida Test Automation Meetup Notes, January 2020
A little bit of history before we jump to January meetup notes. About four years ago I got a chance to lead South Florida Test Automation Meetup, together with Philip Daye we started organizing…
Jan 7, 2020 · 3 min read
Do we have to do UI automation for everything in 2020?
Lately, we more and more hear that not every UI test needs to be automated. I tend to disagree with this statement. Even though there are cases where automating something will take longer than…