How does StarCoder-15b generates concept, instruction, and responses?

[wasiahmad]

Is StarCoder-15b an instruct model itself? If not, how does it is utilized in concept, instruction, and response generation?

[UniverseFly]

Hi, StarCoder-15b is a base model, and we use it to generate data and create starcoder2-instruct. The main pipeline code is here: https://github.com/bigcode-project/selfcodealign/blob/46749cceeb84cf1773deb5c157b04930acebc5cd/src/star_align/self_ossinstruct.py.

[wasiahmad]

Thank you for getting back to me. I saw the code. Given a code snippet, isn't it hard to generate concepts/instructions for a base model? I saw the code and realized the prompt was quite simple and straight-forward. Did you evaluate how well the base model in generating concepts/instructions? Any analysis or findings from earlier experiments?

[UniverseFly]

You can find the instructions from an earlier selfcodealign, starcoder2-instruct. Sorry the instructions for the new version has not been updated yet. It's mostly compatible with the old one, but we will release an updated README soon.

Regarding the analysis, the section 4.1 (Self-Alignment with Different Models) in our paper might be of your interest.