Proposal: TezTok integration

Community
, , ,
1

Hey Teia-Community,

some of you probably know that, besides participating in the Teia community, I’m also working on a project called TezTok. The goal of TezTok is to make it easier for developers to work with Tezos NFTs and to enable them to build great user experiences. The core of TezTok is Open Source (licensed under MIT) but we are aiming to build some commercial services on top of it at some point.

For the TezTok project, we are currently in the process of re-applying for a grant at the Tezos Foundation. A few months ago we already applied once, but they asked us to rework the request, go down with the numbers and be more specific about certain deliverables. Since we are in a bear market now, the Tezos Foundation seems to be gotten a bit more conservative about funding projects.

Anyway, I was thinking if it would make sense to add some Teia-specific topics to this grant request, which I wanted to discuss with you.

The core idea is to migrate Teia to use the TezTok indexer. As of now, the TezTok indexer already supports about 80% of the functionality of what Teia needs. This includes things like token metadata data, sales, listings and a full token history. What it currently doesn’t support are collab contracts and subjkts. Adding those missing features would be covered by the grant.

Just like it’s currently the case, the indexer can be hosted on community servers, like the ones operated by Manitcor and Zir0h.

What are the benefits of migrating to the TezTok indexer (obviously, I’m a bit biased on the topic)?

  • The indexer supports the generation of thumbnails (images, videos). Using these thumbnails on overview pages will significantly improve the loading performance. If needed, thumbnail generation can happen on a separate server.
  • The indexer stores things like the current cheapest price and the last sales price on the token model. This means it’s trivial to output these kinds of information on overview pages.
  • The indexer also stores marketplace data of other marketplaces such as objkt.com and Versum. This means if we show pricing information on overview pages, this could involve the prices of other marketplaces that support Teia NFTs.
  • The indexer stores a full history of token-related events. What this means is, that we would be able to show a more detailed history for each token. Including things like token transfers, cancel swaps, objkt.com sales/offers, etc.
  • Sales-related data is stored in a way that makes it easily aggregatable, making it trivial to answer questions like: How many sales did a certain artist have in total? How much did a user spend on Teia NFTs in a certain time period? Which marketplace contract do users prefer to buy a Teia NFT?
  • If Teia introduces a new FA2 contract and/or a new marketplace contract in the future, it’s easy to add support for those (the whole point of the TezTok indexer is to support a variety of different NFT-related contracts and normalize the data).
  • I’m actively working on the indexer. Every new feature I add will also be available to the Teia Community.
  • Although I still come across some bugs here and there, the indexer is working very reliable in general. In fact, the indexer is running with nearly no interruption for several months now.
  • Re-indexing can happen while the indexer is running in production. This means, no more downtime because we have to do a full re-index.
  • The indexer is very fast and can handle chain reorgs just fine. In practice, this means, transactions like sales, swaps and mints will be reflected in the UI without huge delays.

Of course, nothing in life comes with only advantages, so here are the disadvantages/risks I can currently think of:

  • The current indexer, hicdex, is built on top of dipdup. Dipdup is a well-established indexer framework that is used in the Tezos ecosystem. It is well supported and has great documentation that guides developers. The TezTok indexer, on the other hand, was built from scratch without any indexer framework for the sole purpose to index NFT-related data. The TezTok documentation is currently lacking, making it harder for third-party developers to contribute. A part of the grant proposal is to ask for funds to improve the documentation.
  • Although the TezTok indexer is licensed under MIT, making it possible for any other developer to contribute, a lot will still be depended on me, because the indexer is my brainchild. If I get hit by a bus or am forced to leave the NFT space for other reasons, it might be a bit more challenging for other developers to do code changes because they are not familiar with all the ins and outs.
  • The TezTok indexer also aggregates a lot of data that is not that relevant for Teia (for example, FXHash tokens). Either this will bloat the database a bit or we will need to find a way to exclude that data.

If the Teia community agrees with me that switching to the TezTok indexer makes sense, then I would put the following points into the grant request:

  • Add support for Teia Collab Contracts and Subjkts to the TezTok indexer.
  • The initial creation of thumbnails for all Teia/HEN NFTs. Optionally uploading them to a CDN like Backblaze.
  • Add TezTok support to the frontend (+ do some refactorings as part of that).
  • Use the generated thumbnails on all overview pages.
  • Show price information on overview pages (e.g. current price, last sales price).
  • Enhance the token history and display further information like cancel swaps, transfers and objkt.com-related events.
  • Some other frontend-related changes that the community needs to decide on?

I would estimate the workload at roughly 20 days, asking the Tezos Foundation for 800$ per day. While the indexer-related changes will likely have to be done by me, we can split the other tasks across multiple developers (if there is interest), meaning that everyone who is involved in this operation will get paid.

If you have any questions about this, feel free to comment in this thread. If you can think of further advantages, disadvantages, or other points for the grant, make sure to point them out.

In order for us to move forward, it would be great if we reach a decision by 17.07.2022.

1 Like
2
  • If you get the grant, will the Teia features be prioritized over other work for TezTok?
  • You mention an estimate of 20 days; I’m assuming that is just the TezTok development time. What is a reasonable timeline for completely replacing the existing Teia indexer?
  • What is the current TezTok hardware setup and how well can it scale?
  • What would the cost be for Teia using the existing TezTok server?
  • What is the cost of running a Teia-specific instance of TezTok?
  • How exactly will the documentation be improved? Are you planning to hire somebody to do that task?
  • What kind of support will be provided if issues arise with the Teztok integration and what is the cost?
3

great questions :slight_smile:

If you get the grant, will the Teia features be prioritized over other work for TezTok?

No, they will have the same priority. The grant request will also contain some teia unrelated stuff (not mentioned in the text above). I’m also in the process of building other products on top of TezTok.

You mention an estimate of 20 days; I’m assuming that is just the TezTok development time. What is a reasonable timeline for completely replacing the existing Teia indexer?

The 20 days estimate involves everything that is needed to completely replace the existing indexer. My rough guess: 7 days of indexer backend work, 5 days to integrate it in the frontend, 1 day to deploy it on a community server and 7 days to implement additional stuff in the frontend (like a better token history, prices on overview pages and what else we come up with).

What is the current TezTok hardware setup and how well can it scale?

I’m using this server here, although it’s probably a bit overkill: https://www.hetzner.com/dedicated-rootserver/ax101

I don’t think it’s very likely that horizontal scaling is needed (Tezos currently has 50,000 active wallets). But if it’s needed, TezTok can split up workload onto multiple servers. For example, there could be some servers for thumbnail generation. They would all receive their workloads from a single postgresdb, which TezTok uses as queuing backend.

That being said, I propose that the Teia TezTok server is not run by me directly, although it would surely help if I would get access to that server.

What would the cost be for Teia using the existing TezTok server?

Good question. I’m currently paying 111 Euro a month for our TezTok server. If Teia would start using that as well, we would probably have to share that cost. But as already mentioned, probably better if Teia uses its own infra to not be completely dependent on me.

We would also need to figure out the thumbnail-related costs.

What is the cost of running a Teia-specific instance of TezTok?

I think a 100 Euro hetzner server should do the job and maybe a second server as backup. It would need be checked if TezTok can also be operated on a virtual machine (never tried).

I currently can’t estimate the full costs of the thumbnail-related stuff. This depends on if they get hosted on a CDN like Backblaze, etc.

How exactly will the documentation be improved? Are you planning to hire somebody to do that task?

We are planning to create a “knowledge center”. This is also a part of the grant request (also not mentioned in further detail in the text above). I’ll be creating the content for the knowledge center, since it requires technical expertise, and my buddy Cliche will do some web design for it. It’s not strictly defined yet what will be part of this documentation, but I’m thinking of describing the data models, all the TezTok events, adding graphql examples, and giving an architectural overview of the code and how it can be extended.

What kind of support will be provided if issues arise with the Teztok integration and what is the cost?

I see my Teia involvement primarily on a community level. This means unless Teia suddenly starts making a lot of money through fees, I’ll just keep on contributing on a volunteer base (with all the pros and the cons of volunteer-based work).

I’m already constantly improving TezTok and fixing stuff. No intention of changing that. As mentioned, we are also planning to build some commercial products on top of it.

4

In response to the points made by noFungible on twitter (see https://twitter.com/noFungibleNFT/status/1545239211745300480).

It seems risky to switch away from a DipDup based platform in favor of a home brewed indexer that only the creator can work on (this is called out in proposal).

I’m not claiming that only I can work on the codebase, just saying that as the person who created the indexer, it’s probably simpler for me to make code changes than it would be for someone who is not that familiar. The same thing can be said for most other codebases as well. If you look through the commit history of, let’s say, dipdup or tzkt, you will also notice that most commits come from single people. I don’t think it’s too far fetched to assume that they are likely the ones most familiar with the code. That being said, I agree that switching indexers doesn’t come without risks.

Not sure how I feel about the term “home brewed”, considering that I spent months working on it. (although i spent most of the time developing on it in my bedroom. So from that perspective, you might be right :wink: )

Additionally, the proposal does not explicitly include refactoring, and documentation authoring, as an attached as a work requirement for adding Teia support as a selling point for them receiving the grant.

That’s right, I didn’t add those here because they will be part of the grant request, regardless if the community decides to go forward with the proposal or not.

Essentially, we are inheriting technical debt, with no guarantee of refactoring.

That’s right, there is no guarantee on that. But do we have the guarantee currently, that someone will keep on improving hicdex? The development on hicdex recently picked up, but before that there was a long period of all sorts of emergencies and instabilities (I know that, because I was one of the people involved doing hicdex related debug sessions on weekends).

Also somewhat of a conflict of interests given that @TeiaCommunity is being utilized to receive grant funds that will undoubtedly be used for proprietary endeavors.

I do think that the Teia part of the grant request might make it a bit more likely to be accepted overall. That being said, the TezTok indexer is also used in a lot of different other projects as well. So although Teia using it would probably be a plus, I’m quite confident that it will also be accepted even if the community votes against the proposal. To be clear, I also won’t be pushing this proposal against all odds. It’s a suggestion, nothing more, nothing less. If I have the feeling that there is big resentment against the proposal, even if it should pass the vote, I probably would withdraw it because it would mean fighting constant uphill battles, which I don’t have a huge interest in :slight_smile:

The whole grant request only contains open source and documentation aspects. But you are right that we are also building some proprietary services on top of it, they are not subjekt to this grant request though (might be in future grant requests though).

Maybe a better use of time would be to build an effort around improving a hicdex fork or starting a new DipDup?

Development/Maintainance on the hicdex fork recently picked up, so I agree that this might also be an option. Would you be willing to contribute?

5

i guess there will still be ppl using hicdex instances esp for tool building/maintaining old tools
thinking about redundancy wouldnt it make sense to still maintain a hicdex instance (like zir0hs .rocks one that was initially set up as a backup/alternative instance? or is that overkill?

6

yes, i think the hicdex instance would need to be kept up for some time. maybe by clearly communicating an EOL after half a year or something.

7

There is a community-wide operation going on, led by Tezos Commons, to solve the thumbnail problem for everyone.

I think under these new circumstances, it doesn’t make a lot of sense to keep these points in the proposal:

  • The initial creation of thumbnails for all Teia/HEN NFTs. Optionally uploading them to a CDN like Backblaze.
  • Use the generated thumbnails on all overview pages.

So I would suggest that we remove them and reduce the overall estimate from 20 days to 17 days.

8

I think the hicdex developer would agree that hicdex was never designed to be used for a production marketplace but rather for third-party developers who wanted to experiment with blockchain data.

1 Like
9

march said that multiple times over the last few weeks, so yes i think we can safely assume that.