Hey all, I started prototyping an idea for a new performance tool and would love your thoughts + advice.
Problem
When I’m working locally, it’s really hard to tell whether my code changes actually made any difference.
Goal
As a developer, I want to see the performance impact of my changes in realtime as I’m actively working on a website.
Current Solutions
- Run multiple Lighthouse tests and compare.
- Manually check DevTools’ Performance tab and compare.
- Deploy to a dev/staging/QA environment and run PSI or WPT multiple times and compare.
Oh, and don’t forget your baseline! Remember to repeat your tests before and after your code changes so you can actually measure the difference… I forget this all the time.
Automated tests on Pull Requests are helpful, but still, the feedback loop is so delayed. I want more realtime feedback.
What other tools already do this? Or at least come close to the idea?
Idea: Monitoring for local development
Here’s a prototype I’ve started hacking on to get some ideas flowing:
- Left side: An example web page with a basic heading, picture, and paragraph. Every time I reload the page, metrics are sent over to the monitoring app on the right.
- Right side: A prototype monitor collects metrics from the page I’m developing and shows them in a line chart where I can easily see the impact of my changes over time.
Source code
If you want to try out what I have so far, here’s the GitHub repo for “perfwatch”: GitHub - tannerhodges/perfwatch: Monitor performance changes during development.
Features I want
- Connection: Easily connect to any website I’m working on. Automatic if possible, minimal setup otherwise.
- File detection: Detect file changes on my local filesystem, so I can see which code changes had what impact. Ideally, I’d like to be able to select a folder and say, “These are the files for my website. Let me see when these files changed in the timeline.” And then in the monitor, I could see the files and code changes annotated somewhere on or below the timeline.
- Notifications: Notify me of any large differences in metrics. For example, if I have the monitor running in the background, I want a push notification that a metric suddenly jumped 3 standard deviations.
- Compare ranges: I want to be able to easily compare two ranges of metrics and report what the difference was. For example, if I start working on a change, I want to mark my first 10 page loads as the baseline, then make my code change, then do another 10 page loads and automatically calculate what % improvement I’ve made. Ultimately, I just want to know was my change significantly better or worse.
- Check current device/connection settings: To help qualify any statements people make using this tool, I’d also like to detect and list somewhere the current environment: What device/OS/browser/connection speed do you currently have? ← Might be interesting to treat this like file changes too and notify when, for example, a drop in connection affects metrics for a particular page load.
I realize this is a lot. (I haven’t even mentioned Storage, Custom Metrics, and Reports as features…)
Challenges
I’m not 100% sure how to make some of these features work. In particular, I feel like the connection and file detection features are my biggest challenges:
- How can this monitor connect to websites? (Injected script? Manual snippet?)
- How can it access and detect local file changes? (File System API? Electron app?)
Anyhow, that’s the idea: local performance monitoring. Wadda ya think?
Appreciate any thoughts, advice, or feedback y’all might have!
Hopefully I can simplify enough to make this indie-feasible.
My kingdom for this on every project.
Selfishly, I think I want an even faster loop…
But confounding is a good callout…