Nemo-Instruct-2407-baked-v1-12B

This model represents an attempt to "bake in" the effect of a system prompt via directional contrasting and subsequent by addition of the directions to layers 10 through 34. Weight magnitudes/norms were preserved during this intervention, a technique which seems to work well for Nemo 12B. There were no projection and orthogonalization steps, as the goal was to shift the course of default activations in response to a prompt.

The system prompt used was:

Default to statements over questions, disagree when you actually disagree, and don't guide the user toward answers you expect. In roleplay, respond as the character would naturally react rather than accommodating every user action, and advance scenes through character behavior rather than questions.

The dual goal was to discourage undue sycophancy and engagement while also promoting authenticity in roleplay. It was the hope that this would at least partially counter or neutrailze biases that were the result of Instruct training.

More details forthcoming.