A note on language and opacity
I write this series in English because Polymarket isn’t accessible in every jurisdiction, and the audience most likely to find it useful is spread across regions where English is the shared technical language. A word on what I do and don’t share: the engineering is described in full, because there’s nothing secret about a memory bug or its fix. The strategy — which signals I follow, how I filter them, how positions are sized — I keep private deliberately. That’s the edge. Everything else is fair game.
The setup
Any bot that acts on real-world events has to solve the same basic problem: the world will deliver the same event more than once, and the bot must not act on it more than once. The network retries a message. A polling loop catches the same record on two consecutive ticks. A process restarts mid-flight and replays from the last checkpoint. All of these produce duplicates, and duplicates that trigger trades are expensive mistakes.
The standard answer is deduplication: keep a record of what you’ve already handled and check incoming events against it before doing anything. If the event is in the record, skip it. If it’s not, process it and add it. Simple, correct, and — as I found out — quietly broken in a way that takes a while to notice.
What was actually happening
The bot maintains two in-memory sets. One tracks identifiers for signals it has already acted on. The other tracks identifiers for on-chain events it has already confirmed. Before doing any real work, the relevant identifier gets looked up in whichever set applies. If it’s present, the code returns early. If it’s absent, the code proceeds and then inserts the identifier.
That logic is correct. The problem was the other half of the lifecycle: eviction. Nothing was ever removed. An identifier went in on first sight and stayed there until the process died. The sets grew monotonically for as long as the bot ran. On a short test run that didn’t matter. On a process running continuously, it accumulated silently until memory pressure or a manual restart cleared it.
The part that stung a little when I found it: there was already a config value called something like dedup_window_seconds. I had written it early on, set a sensible-sounding value, and then completely failed to wire it up anywhere in the code. It sat in the config file doing nothing. The intention had been there from the start. The implementation had not.
# config.yaml — value present, never read
dedup_window_seconds: 90
# what the code actually did
seen_signal_ids = set()
def handle_signal(signal_id, ...):
if signal_id in seen_signal_ids:
return # skip duplicate
seen_signal_ids.add(signal_id)
# ... do the real work
No expiry, no eviction, no use of that config field. The set never forgot anything.
Why bounded deduplication is the right model
It’s worth being precise about what deduplication is actually trying to prevent. The risk is acting on the same signal twice in quick succession — a retry arriving a few seconds after the original, a polling tick catching a record that the previous tick already handled. That window is short. A signal from several minutes ago is not a duplicate risk; it’s just history. The market has moved, the moment has passed, and any repeat of that identifier at that point is almost certainly a genuinely new event or an edge case that shouldn’t be silently swallowed anyway.
Deduplicating against something from hours ago serves no purpose. It protects against nothing real, and it costs memory continuously. The correct model is a sliding window: an entry is retained for long enough to cover the actual retry and replay scenarios, then evicted. “Have I already handled this?” should mean “in the last N seconds,” not “at any point since launch.”
The fix
Instead of a plain set, each entry now stores a timestamp alongside the identifier. On every insertion, old entries are evicted first. The eviction threshold is read from the config field that had been waiting unused.
import time
from collections import OrderedDict
class BoundedDeduplicator:
def __init__(self, window_seconds):
self.window = window_seconds
self._store = OrderedDict()
def _evict(self):
cutoff = time.monotonic() - self.window
while self._store:
oldest_key, oldest_ts = next(iter(self._store.items()))
if oldest_ts < cutoff:
del self._store[oldest_key]
else:
break
def seen(self, key):
self._evict()
return key in self._store
def mark(self, key):
self._evict()
self._store[key] = time.monotonic()
self._store.move_to_end(key)
Usage at the call site is unchanged in shape — check, then mark — but now both operations carry the eviction step, so the store stays bounded without a separate background thread or scheduled task.
deduplicator = BoundedDeduplicator(
window_seconds=config["dedup_window_seconds"]
)
def handle_signal(signal_id, ...):
if deduplicator.seen(signal_id):
return
deduplicator.mark(signal_id)
# ... do the real work
The OrderedDict insertion order gives us the eviction loop cheaply: entries are always ordered oldest-first because we insert in arrival order. Walking from the front and stopping at the first non-expired entry is O(k) where k is the number of expired entries, not O(n) over the whole store.
The restart property
There's a second benefit that I hadn't fully thought through until after the fix was in. With unbounded sets, a process restart was a meaningful event from a correctness standpoint: it cleared all deduplication state. Restarting was the only way to reclaim memory, but it also meant that anything in-flight during the restart window could be replayed and double-processed. The two concerns were in tension — you needed to restart for memory hygiene, but every restart was a small correctness risk.
With a bounded window, restart risk shrinks to almost nothing. On restart, the in-memory store is empty, which means the bot will reprocess any identifiers it saw in the last dedup_window_seconds before the shutdown. That's acceptable — it's exactly the same window you'd be vulnerable to in any crash scenario — and it's bounded. You're not vulnerable to replaying events from three hours ago because those events wouldn't have been in the store anyway.
This reframing matters for how you think about the process. A process with unbounded dedup state is stateful in a way that makes it fragile to restart. A process with a short bounded window is effectively stateless across anything longer than that window. Deploy it, restart it, crash it — the behavior converges to correct quickly. Idempotency stopped being an accident that depended on the process never dying and became a property I could reason about and rely on.
The config field lesson
The unused config value is the part I keep coming back to. Writing a config field is easy. It feels like progress. But if the field is never read, it's documentation at best and misleading at worst — it implies the system has a capability it doesn't have. I've started treating unread config fields the same way I treat dead code: they should either be wired up or deleted. Leaving them in place creates a false map of the system.
In this case the field sat there for long enough that I genuinely forgot I'd never used it. Finding it during the memory investigation was useful — it meant I had a config-first path to the fix rather than inventing a new mechanism — but the better outcome would have been catching the disconnection at the time of writing. A simple grep for config keys in the codebase, run occasionally, would have surfaced it. I've added that to the maintenance checklist.
Where this leaves things
Memory is now stable across long runs. The deduplication window is narrow enough to catch all realistic retry and replay scenarios and wide enough to not miss anything that arrives with normal network latency. Restarting the process is routine again rather than something that requires thinking about timing. The infrastructure is a little more honest about what it actually does.
Next up: what happens when the upstream data source goes quiet — distinguishing a genuine market lull from a silent connection failure.