The label was the whole experiment

The decision I was trying to automate

I run a copy-trading stack on FXVPS. A master account emits signals, and every week I sit down with a P&L audit and make the same call on each strategy: take it as is, invert it, or skip it. Some strategies are doing exactly what they should, and their signals are worth following in the original direction. Others are consistently wrong in a way that makes them valuable reversed. The rest are not worth copying in either direction.

Written down like that, the decision sounds trivial: three outcomes, one choice. In practice, the judgement behind it is anything but. A weak month does not make a strategy bad, a losing streak does not mean I should flip it, and a healthy win-rate does not mean it is actually making money. Every week I have to look past the surface result and work out what the strategy is really doing. The audit works, but it is manual and it lives in my head, so I started building a model to make the same call. I was not trying to replace the judgement with a black box; I wanted to see whether the decision I was already making could be defined clearly enough for a model to learn it. I assumed the hard part would be the modelling. It was not: the first real problem showed up before I trained anything.

The setup that feels obviously right

The natural way to frame this is as a classification problem. You pull the MetaTrader5 history, label every completed trade a win or a loss depending on whether it made money, train a classifier to predict which trades will win, and use those predictions to decide whether to follow a strategy or reverse it. The logic is clean: winning trades are good, losing trades are bad, and a strategy the model expects to lose should be inverted. It turns a messy judgement into a tidy binary target, and that tidiness is seductive: once every trade is a 1 or a 0, the whole pipeline falls into place and pandas and scikit-learn can take it from there.

But a clean target is not the same thing as a correct one, and the moment I checked the label against how the strategies actually behave, the setup fell apart. These are asymmetric trend strategies: they take many small losses and occasionally catch a large win, and that shape is the entire point. Across roughly 9,500 real trades the overall win-rate is about 32%. Treated as an objective, that number looks dismal: a classifier trained to maximise wins would stare at a sea of losing trades and conclude the signal is usually wrong. But the 32% is by design, not a defect. These strategies do not need to win often; they need their winners to outweigh their losers, and a strategy can lose again and again, catch one big move, and still come out well ahead. The win/loss label erases exactly that: a tiny win and a huge win both become a 1, a trivial loss and a painful one both become a 0. The label was clean, and it had quietly flattened the economics of the entire book.

The strategy that gave it away

It became impossible to ignore when I looked at one strategy in particular. It won just 32.7% of the time, which under the win/loss framing made it an obvious candidate for inversion: it loses more than it wins, so a model trained on those labels would learn to bet against it. But that strategy was, in fact, the most profitable one in the book: 150 trades, a profit factor of 1.24, and a positive expectancy in points while most of its higher win-rate neighbours were quietly losing money. That was the moment the experiment changed. The model had not made a bad prediction; I had not even got that far. The target itself was telling the model that my best signal was bad.

Had I carried on, the model could have trained cleanly, produced sensible-looking metrics, and still handed me the wrong decision. It would have learned, correctly, that any given trade from this strategy was more likely to lose than win, a true statement, and the wrong question. I did not need to know whether the next trade would close green; I needed to know whether following the strategy had positive expected value, whether reversing it did, or whether neither was worth the risk. Those are different questions, and a model optimised for win probability does not just miss the answer to mine, it can confidently recommend destroying value. That is the dangerous kind of failure, because everything downstream looks respectable: the code is correct, the metrics move, the predictions match the labels. The experiment is simply pointed at the wrong thing. The label was not an implementation detail. The label was the whole experiment.

Rebuilding the target

Once I saw it, I had to put the classifier down and go back to the actual decision. What was I really trying to optimise? Not win-rate, not the count of profitable trades, not whether a single trade finished above zero. The target that matches the decision is expected value, measured in points, or price distance, because points preserve the movement a strategy actually captured or gave back, and let me judge the quality of a signal without tangling it up in how much I happened to bet on it.

That last part matters more than it sounds. I could not measure the target in dollars, because dollars are distorted by position size: one trade can earn more simply because it was sized larger, and another can lose less because it was sized cautiously, neither of which tells me which underlying signal was better. A well-sized loser and an oversized winner can swap places when you rank them by dollar P&L. So I re-measured the whole history in points. Instead of asking whether each trade won or lost, I looked at the price distance it produced, which let the target reflect both sides of the asymmetric distribution: the frequent small losses and the rare large wins, rather than collapsing them together. The large winners stopped being compressed into the same token as everything else, the small losses stopped counting as equal evidence against the strategy, and the target finally matched the shape of what I was measuring. More to the point, it matched the decision. Take, invert, or skip was never really about win probability; it is a question about expected value in each direction. If the original signal has positive expected value, take it; if reversing it is stronger, invert it; if neither clears the bar, skip it.

The check that mattered

Changing the target was only half the job. I still had to know whether it was measuring what I thought it was. The useful test was not whether the data looked cleaner or some metric ticked up; it was whether the new target agreed with what I already knew from running the stack by hand. I had ground truth: from the weekly P&L audit I knew which strategies were durable winners and which were only useful reversed. When I re-measured the same MetaTrader5 history in points, those distinctions came back out on their own: the strategies I knew were reverse legs showed up as reverse legs, and the ones I knew were durable winners showed up as durable winners.

That agreement is what convinced me. It does not prove the model will generalise, and it does not finish anything (this is still a build in progress), but it shows the target is finally pointed at the right thing. The win/loss label disagreed with the operating reality of the stack; it wanted to invert my most profitable strategy. The points-based target recovered the structure I already trusted from practice. Before you ask a model to say something about the future, the least it can do is describe the world you already understand, and if it cannot reproduce the obvious calls in the history, there is no reason to believe what it tells you about anything new.

Where the work actually lives

The false start changed how I think about the rest of the project. I had been treating it as a modelling problem: I had the history, a decision to automate, and the usual tools, and the pull was to get to training quickly. But the work that actually mattered happened before any of that: deciding what a good outcome meant, separating signal quality from position size, making the target respect the asymmetric shape of the strategies, and checking the whole thing against decisions I already trusted. None of that is glamorous and none of it involves a clever model. It is framing, measurement, and a refusal to accept a convenient label just because it makes the dataset easier to train on.

That is exactly why it is the valuable part, and why it sits upstream of the model rather than inside it. A sophisticated model cannot recover information the target has already thrown away: it cannot know a rare large win matters more than a handful of small losses once both are 1s and 0s, and it cannot tell a good signal from a big position once the label has blended them into dollars. The target is what tells the model which reality to care about. Get it wrong, and better modelling only makes the wrong answer more convincing.

The takeaway

Most of the time, "the model did not work" really means "the label did not mean what I thought it meant." The failure starts before features, training, or tuning; it starts when a real business objective gets translated into a target that is easy to compute but does not represent the decision you are actually making. Here, win/loss looked obvious and was wrong. The 32% win-rate was not a sign the strategies were broken, it was how they worked. The 32.7%-win strategy was not weak, it was the best I had. Expected value in points captured that, win/loss did not, and dollars could not, because dollars were distorted by position size; once I re-measured in points, the history reproduced the reverse legs and durable winners I already knew.

So the lesson I am carrying into the rest of the build is boring and load-bearing: decide what you are optimising before you touch a model, define it in the units the real decision is made in, and test the target against ground truth before you trust anything downstream. The hard, valuable part of applied ML lives upstream of the model, in the target, the leakage discipline, and the metric. The modelling is the easy mile.