Differences

This shows you the differences between two versions of the page.

--- onetime:babak [2025/01/20 16:21] – tomas
+++ onetime:babak [2025/01/24 07:58] (current) – tomas
@@ Line 2: / Line 2: @@
 this is an overview of projects that I want to do in the context of our recent cooperation. This is aimed as a base for discussion about the scope and budget, possibly we may find a way to cooperate further.
+[[babak_reasons|Here is a page]] describing the reasons why we want exaxtly that what is in this page.
 ====== Goal ======
@@ Line 25: / Line 27: @@
 The result of this part of work would be the script(s), not the data actually because we then will want to repeat the same process with other scraped data.
 ) The vectors should be uploaded to a vector database. We would like to use Pinecone over the API. We don't have yet established any infrastructure in that DB.
 The task for this part would be to write a PHP script that takes the directory with the embedding files, see point (2) and uploads them into Pinecone (creates a new namespace in a given index, the script would get the API keys and other such data as an input).
 You would test the script with your DB and your OpenAI API and then give it to us along with some documentation what we need to do to prepare the whole scenario to work. Alternatively we could do it so that you use our DB and our keys and we do the preparation before that based on your guidance.
@@ Line 36: / Line 38: @@
 The script would create 2 namespaces in Pinecone, one containing main vector -> "0000123" and the other containing 4 vectors all pointing at "0000123". If some search matches any of these vectors, we receive the ID "0000123" and we know which article was found by that search.
 ) Write a PHP script that takes a text on the input, embeds it into a vector using OpenAI API and finds a given number of best matches from the given Pinecone namespace using Pinecone API.
 ) Scrape the bigger data (doctors answer questions at https://www.ulekare.cz/poradna-lekare). These are 230k webpages. We need to parse these ones a little differently than in point 1 - not creating any paragraphs and captions but only have 3 parts: 1 question and 1-2 answers, separated as 3 HTML paragraphs separated by captions so that we can make sure the embedding script will embed the parts separately.
 ) Now for embedding this data we would like to use the same script from point 2. That script expects that the first vector is created from the whole text and the rest is the subparts. For processing this part of data we would need to omit the first vector and only create the sub-vectors. This way the uploading script (3) will separately upload the first sub-vector (i.e. the quesion) and searately the answer(s). The script in 2 must be updated to allow for optional creation of the first vector from the whole article.
+) All of this will incur costs for the underlying services - embedding API, vector DB usage. The idea is that the vendor of the scripts bears these costs for testing purpose needed to produce the scripts and then we bear the costs for the production usage of the scripts. In both cases we would need to know in advance what costs these will approximately be because the projects specs will need to include some max covered costs specification.
+) This description basically outlines the program for several PPH projects but it is NOT yet full project specification. The projects have some unclear points and need to be discussed. The definitive form of the PPH projects includung the negotiated price must yet be derived from this description. The vendor is expected to cooperate with us on delivering this prior to starting on the work in PPH.
+UPDATE 10) We need the PHP scripts to use only http reqiests, not any more complex libs. Reason see [[babak_reasons|reasons page]].
 Last) In case you have any thoughts about the solution or suggestions how this could be improved or changed from some good reason, we can discuss.
@@ Line 51: / Line 59: @@
 ) Already done.
-) Embedding scripts. Will reuquire to do the splitting part smartly plus call the API. Expected budget 250 euro (plus the API service costs).
+) Splitting scripts. Expected budget 100 euro.
+) Embedding scripts. Should be straigtforward, expected budget 80 euro (plus the API service costs).
+) Script for Pinecone upload. Similar stuff but more complex, 150 euro (plus the API service costs).
-) Script for Pinecone upload. Similar stuff but less complex, 150 euro (plus the API service costs).
+) Finding matches from Pinecone - this should be easy, 70 euro.
-) Finding matches from Pinecone - this should be easy, 70 euro.
+) Scrape the bigger data. This is 220k pages that need to be scraped similar like you did (1). You offered a price of 1600+ euro, that's way too much for us. We could offer a price of 800 euro for this task. Couldn't you find a way how to do it for that money?
-) Scrape the bigger data. This is 220k pages that need to be scraped similar like you did (1). You offered a price of 1600+ euro, that's way too much for us. We could offer a price of 800 euro for this task. Couldn't you find a way how to do it for that money?
+) This is really a part of 2, I only explained it separately for better clarification.
-) This is really a part of 2, I only explained it separately for better clarification.
+and 9) These are points that must be determined before we start on any of the projects.
 I would like to discuss this plan with you. If you find a way how to do some parts of it with the suggested budgets, then we could define this as PPH projects and award them to you directly.
-I would prefer you as you delivered high quality result but the budget is very important for us as well so possibly I would need to try getting it done by other people. Expecting your comments.
+I would prefer you as you delivered high quality result but the budget is very important for us as well so possibly I would need to try getting it done by other people. Expecting your comments.77