.hgsubstate not updated on "pull -u" or "update"

[wrmarvin]

Using HG version 2.4 on windows XP.

I have a "Master" repository of source code on a server at a company's site.
I am the only programmer currently working on their software. The master
repositories do not have working directories. I work on this software from 2
locations, my office (I am a consultant with VPN access to their server),
and on a workstation at their office. They have multiple VB6 projects (52
projects) that create .exe or .dll executables that share source code (both
.frm and .bas files).

There are repositories for shared code (40 repositories) and a project
repository for each project (52 repositories). The project repository for
each project contains subrepos for the shared code modules that it needs (0
to 10 subrepos in each project). The 40 shared code repos and the 52
project repos are all directories located at the same level in the directory
tree, so the .hgsub file in the project repo uses relative referencing like
"SubSharedCodeRepo1 = ..\SharedCodeRepo1".

I use the following workflow:
1a) clone a project (Project1) to be working on at location A (e.g. my
office) from the "Master"
1b) make changes in primary repository files and/or shared code files in
subrepos locally.
1c) commit any subrepos with changes
1d) commit primary project repo
1e) push primary repository back to the "Master" (this pushes any modified
and commited subrepos)
1f) Note: project repository (Project1) at location A site is not deleted
after push.

2a) Go to location B (company office)
2b) clone the project (Project1) from "Master" on the server to a
development computer.
2c) make changes in shared code in 1 or more subrepos.
2d) commit all changed subrepos
2e) commit primary project repo (this changes .hgsubstate to the just
commited subrepo(s))
2f) push the primary project repo (Project1) back to "Master" while at
location B.

3a) Go back to location A
3b) cd to be within the local existing project repository (Project1).
3c) "pull -u" all subrepos used by the project from "Master" (pull will not
recurse subrepos, so subrepos must be pulled first individually) (TortoiseHG
will recurse pull of subrepos during a synchronize, but for this test I only
used mercurial hg commands. The same problem happens when using TortoiseHG
synchronize)
3d) "pull -u" primary project repo.
3e) AT THIS POINT: .hgsubstate did not get updated from what is in the tip
of the primary repository.
3f) If you try a manual "update -rtip" it still does not update the
.hgsubstate file.
3g) If you do a kdiff between the working directory .hgsubstate and the
repository tip .hgsubstate it will show that the files are different.

I would expect the "pull with update" action in 3d to put the project
working directory files to be the same as the files in the tip of the
project repository. Doing a kdiff on any tracked file in the updated working
directory should show no differences to the files in tip and that includes
the .hgsubstate file. If I do a full clone, the .hgsubstate file DOES become
the same as the file in tip.

There could be a problem if you do not pull all the subrepos before pulling
the primary repository. If you pulled the primary repository first and the
.hgsubstate was updated to what is in tip, the .hgsubstate could refer to a
subrepository that does not yet contain the referenced changeset in the
updated .hgsubstate.

If you do the "pull -u" commands in the correct order, the .hgsubstate
SHOULD get updated in the working directory for the project.

For the pull -u command on the primary to work correctly, a check of the
.hgsubstate in the tip would have to be made to make sure that the
changesets referenced in the .hgsubstate do actually exist in the local
subrepos. If all changeset references for the subrepos are valid, then the
.hgsubstate should be update in the working directory of the project. If
there is a changeset reference error, then the .hgsubstate should not be
updated and an error message created.

Even if all all the references are valid, the .hgsubstate from the project
tip could reference a changeset that is not the tip in the just pulled
subrepo. If that is the case, then the working directory of the subrepo now
contains code that was not used for this project when the last push for this
project was done. This can happen because the shared code was changed,
committed, and pushed from another project that uses this shared code.
Usually in ongoing development, I would want to use the newest shared code
for a new build and this is OK. If this situation is detected during a pull
of the project repository, a warning message would be nice so I can then
decide to either use the newer code or revert back to the changeset that was
last used by this project. If I am trying to look at code from an older
version, I would do a full clone of that project version so that I would get
the code in the subrepos that existed at the time of the project build.

After doing the pulls in the correct order, I have to do a "hg revert -rtip
.hgsubstate" in the primary project directory to make the project files the
same as those contained in the tip.

The reason I want to do the pulls instead of a new clone is because location
A and B could be "out of sync". If I delete the project repository at
Location A and clone during section 3 above, I could loose file
modifications or changes at location A (commited) that I failed to push
before going to location B. By pulling, I can then fix things and merge the
heads.

I did not create this mess of projects and shared code, I just now try to
maintain it. The original programmer is gone from the company. I have been
supporting it and using these mercurial repositories for about the last 2
years. I have not had an issue with the .hgsubstate not getting updated on a
pull or update, but it seems that mercurial is not handling this situation
as it should.




--
View this message in context: http://mercurial.808500.n3.nabble.com/hgsubstate-not-updated-on-pull-u-or-update-tp3995591.html
Sent from the General mailing list archive at Nabble.com.

[Matt Mackall]

Post by wrmarvin
Using HG version 2.4 on windows XP.
I have a "Master" repository of source code on a server at a company's site.
I am the only programmer currently working on their software. The master
repositories do not have working directories. I work on this software from 2
locations, my office (I am a consultant with VPN access to their server),
and on a workstation at their office. They have multiple VB6 projects (52
projects) that create .exe or .dll executables that share source code (both
.frm and .bas files).
There are repositories for shared code (40 repositories) and a project
repository for each project (52 repositories). The project repository for
each project contains subrepos for the shared code modules that it needs (0
to 10 subrepos in each project). The 40 shared code repos and the 52
project repos are all directories located at the same level in the directory
tree, so the .hgsub file in the project repo uses relative referencing like
"SubSharedCodeRepo1 = ..\SharedCodeRepo1".
1a) clone a project (Project1) to be working on at location A (e.g. my
office) from the "Master"
1b) make changes in primary repository files and/or shared code files in
subrepos locally.
1c) commit any subrepos with changes
1d) commit primary project repo
1e) push primary repository back to the "Master" (this pushes any modified
and commited subrepos)
1f) Note: project repository (Project1) at location A site is not deleted
after push.
2a) Go to location B (company office)
2b) clone the project (Project1) from "Master" on the server to a
development computer.
2c) make changes in shared code in 1 or more subrepos.
2d) commit all changed subrepos
2e) commit primary project repo (this changes .hgsubstate to the just
commited subrepo(s))
2f) push the primary project repo (Project1) back to "Master" while at
location B.
3a) Go back to location A
3b) cd to be within the local existing project repository (Project1).
3c) "pull -u" all subrepos used by the project from "Master" (pull will not
recurse subrepos, so subrepos must be pulled first individually) (TortoiseHG
will recurse pull of subrepos during a synchronize, but for this test I only
used mercurial hg commands. The same problem happens when using TortoiseHG
synchronize)
3d) "pull -u" primary project repo.
3e) AT THIS POINT: .hgsubstate did not get updated from what is in the tip
of the primary repository.
3f) If you try a manual "update -rtip" it still does not update the
.hgsubstate file.
3g) If you do a kdiff between the working directory .hgsubstate and the
repository tip .hgsubstate it will show that the files are different.
I would expect the "pull with update" action in 3d to put the project
working directory files to be the same as the files in the tip of the
project repository.

...

Does the working directory claim to be clean according to status and
summary before this step?

[wrmarvin]

I am testing this problem and suggestions on a set of test repositories and 2
clones on my local PC. I went back and deleted both working clones. I then
created a new clone in both working 1 and working 2. I made changes in
working 1 in the subrepo and the primary repo. Committed the sub then the
primary. Pushed working 1 back to the "Master" repositories.

I change to the working 2 clone and proceeded as follows:
cd to subrepo and pull (no -u).
check primary status and all file are clean.
hg update subrepo working directory.
hg log of subrepo shows tip changeset is 5:52584d265d5d
cd to primary file in working 2.
check primary status and all file are clean.
look at .hgsubstate in notepad:

05dd8f74ee2be13bd634b3bbd4fb1e1073263d88 SubSharedCodeRepo

This corresponds to changeset: 4:05dd8f74ee2b (subrepo tip-1) as expected
since I have not pulled and updated the primary repo.

next pull (no -u) primary repo.
hg log of primary shows that tip is now correct to tip in the "Master" repo.
"hg status -A" shows all primary files are clean.
open .hgsubstate in notepad and the reference shown above is unchanged (I
have not done the update).
Now update the primary repo working directory to the tip.
"hg status -A" on primary repo shows "M" for .hgsubstate
open .hgsubstate in notepad:

05dd8f74ee2be13bd634b3bbd4fb1e1073263d88 SubSharedCodeRepo

The file is the same as it was before the update and is referencing tip-1 of
the subrepo even though a check of the subrepo parents returns:
5:52584d265d5d.

Now in primary repository do "hg revert -rtip .hgsubstate"
look at status now shows "C" for .hgsubstate
open in notpad shows:

52584d265d5db9e88698ca9c9d522c425e3c335e SubSharedCodeRepo

This corresponds to 5:52584d265d5d which is the parent of the subrepo, and
most importantly, is now the same .hgsubstate file that is in the tip of the
project in the "Master" repository. Now, after the revert of the
.hgsubstate file, my files in working 2 are exactly the same as the files in
working 1 when the push from working 1 was done. Also, now after the
revert, the files are the same as if I make a Working 3 directory and clone
from the "Master".

With further test, I have found an even bigger problem. If I do not to a
"hg revert -rtip .hgsubstate" after the "hg pull -u" a big problem can
happen. My sub repo code is updated to the latest code. I do not have to
make changes to the code in the subrepo, but only make changes in the
primary repo code, recompile the project with the new primary and subrepo
code. Since I did not make any changes in the subrepo code, the status shows
clean, so I do not do a commit in the subrepo. I do a commit in the primary
and then look at the status of the primary. The .hgsubstate is still in "M"
status after the commit. I just compiled with the subrepo working directory
at tip (5:52584d265d5d). Opening .hgsubstate shows:

05dd8f74ee2be13bd634b3bbd4fb1e1073263d88 SubSharedCodeRepo

This is 4:05dd8f74ee2b (subrepo tip-1)

Now I push the project. No errors are reported.
I go to the "Master" repository and extract the .hgsubstate file from the
tip using revert (the Masters have no working directory). Looking at this
.hgsubstate it shows:

52584d265d5db9e88698ca9c9d522c425e3c335e SubSharedCodeRepo

This is correct (5:52584d265d5d) for the state of the repo when I did the
compile, commit, and push, so at the "Master" repository level, all look OK,
but my local working copy of .hgsubstate is still not showing the correct
substate. I would think that during the commit of the primary repo, the
.hgsubstate in the working directory would get updated to the parent
changeset of the subrepo. Since it seems to fix itself on the push back to
the "Master", I guess all is OK. It just does not seem like the timing of
the modification to .hgsubstate on pull -u and commits is what I would
expect.





--
View this message in context: http://mercurial.808500.n3.nabble.com/hgsubstate-not-updated-on-pull-u-or-update-tp3995591p3995623.html
Sent from the General mailing list archive at Nabble.com.

[Scott Palmer]

Do you mean
subrepo = ../subrepo

Using a forward slash atypical of URLs and most operating systems, not the backwards Windows slash. I think I read in the subrepo docs that the standard forward slashes must be used as the .hgsub file needs to work everywhere.

Scott

Post by wrmarvin
Using HG version 2.4 on windows XP.
I have a "Master" repository of source code on a server at a company's site.
I am the only programmer currently working on their software. The master
repositories do not have working directories. I work on this software from 2
locations, my office (I am a consultant with VPN access to their server),
and on a workstation at their office. They have multiple VB6 projects (52
projects) that create .exe or .dll executables that share source code (both
.frm and .bas files).
There are repositories for shared code (40 repositories) and a project
repository for each project (52 repositories). The project repository for
each project contains subrepos for the shared code modules that it needs (0
to 10 subrepos in each project). The 40 shared code repos and the 52
project repos are all directories located at the same level in the directory
tree, so the .hgsub file in the project repo uses relative referencing like
"SubSharedCodeRepo1 = ..\SharedCodeRepo1".
1a) clone a project (Project1) to be working on at location A (e.g. my
office) from the "Master"
1b) make changes in primary repository files and/or shared code files in
subrepos locally.
1c) commit any subrepos with changes
1d) commit primary project repo
1e) push primary repository back to the "Master" (this pushes any modified
and commited subrepos)
1f) Note: project repository (Project1) at location A site is not deleted
after push.
2a) Go to location B (company office)
2b) clone the project (Project1) from "Master" on the server to a
development computer.
2c) make changes in shared code in 1 or more subrepos.
2d) commit all changed subrepos
2e) commit primary project repo (this changes .hgsubstate to the just
commited subrepo(s))
2f) push the primary project repo (Project1) back to "Master" while at
location B.
3a) Go back to location A
3b) cd to be within the local existing project repository (Project1).
3c) "pull -u" all subrepos used by the project from "Master" (pull will not
recurse subrepos, so subrepos must be pulled first individually) (TortoiseHG
will recurse pull of subrepos during a synchronize, but for this test I only
used mercurial hg commands. The same problem happens when using TortoiseHG
synchronize)
3d) "pull -u" primary project repo.
3e) AT THIS POINT: .hgsubstate did not get updated from what is in the tip
of the primary repository.
3f) If you try a manual "update -rtip" it still does not update the
.hgsubstate file.
3g) If you do a kdiff between the working directory .hgsubstate and the
repository tip .hgsubstate it will show that the files are different.
I would expect the "pull with update" action in 3d to put the project
working directory files to be the same as the files in the tip of the
project repository. Doing a kdiff on any tracked file in the updated working
directory should show no differences to the files in tip and that includes
the .hgsubstate file. If I do a full clone, the .hgsubstate file DOES become
the same as the file in tip.
There could be a problem if you do not pull all the subrepos before pulling
the primary repository. If you pulled the primary repository first and the
.hgsubstate was updated to what is in tip, the .hgsubstate could refer to a
subrepository that does not yet contain the referenced changeset in the
updated .hgsubstate.
If you do the "pull -u" commands in the correct order, the .hgsubstate
SHOULD get updated in the working directory for the project.
For the pull -u command on the primary to work correctly, a check of the
.hgsubstate in the tip would have to be made to make sure that the
changesets referenced in the .hgsubstate do actually exist in the local
subrepos. If all changeset references for the subrepos are valid, then the
.hgsubstate should be update in the working directory of the project. If
there is a changeset reference error, then the .hgsubstate should not be
updated and an error message created.
Even if all all the references are valid, the .hgsubstate from the project
tip could reference a changeset that is not the tip in the just pulled
subrepo. If that is the case, then the working directory of the subrepo now
contains code that was not used for this project when the last push for this
project was done. This can happen because the shared code was changed,
committed, and pushed from another project that uses this shared code.
Usually in ongoing development, I would want to use the newest shared code
for a new build and this is OK. If this situation is detected during a pull
of the project repository, a warning message would be nice so I can then
decide to either use the newer code or revert back to the changeset that was
last used by this project. If I am trying to look at code from an older
version, I would do a full clone of that project version so that I would get
the code in the subrepos that existed at the time of the project build.
After doing the pulls in the correct order, I have to do a "hg revert -rtip
.hgsubstate" in the primary project directory to make the project files the
same as those contained in the tip.
The reason I want to do the pulls instead of a new clone is because location
A and B could be "out of sync". If I delete the project repository at
Location A and clone during section 3 above, I could loose file
modifications or changes at location A (commited) that I failed to push
before going to location B. By pulling, I can then fix things and merge the
heads.
I did not create this mess of projects and shared code, I just now try to
maintain it. The original programmer is gone from the company. I have been
supporting it and using these mercurial repositories for about the last 2
years. I have not had an issue with the .hgsubstate not getting updated on a
pull or update, but it seems that mercurial is not handling this situation
as it should.
--
View this message in context: http://mercurial.808500.n3.nabble.com/hgsubstate-not-updated-on-pull-u-or-update-tp3995591.html
Sent from the General mailing list archive at Nabble.com.
_______________________________________________
Mercurial mailing list
http://selenic.com/mailman/listinfo/mercurial

...

[wrmarvin]

Mercurial on a Windows system seems to handle the backslash vs slash either
way. It has no problem referencing the subrepo on a push of the primary or
anything else concerning working with subrepos. On a commit of the primary
mercurial uses the .hgsub as I have written it to reference the subrepo and
updates the .hgsubstate file during the commit to reference the correct
changeset of the subrepo.

I don't think it will make any difference, but I will try changing the
.hgsub to be "SubSharedCodeRepo1 = ../SharedCodeRepo1" and run a test.




--
View this message in context: http://mercurial.808500.n3.nabble.com/hgsubstate-not-updated-on-pull-u-or-update-tp3995591p3995621.html
Sent from the General mailing list archive at Nabble.com.

[wrmarvin]

I went back and changed my .hgsub to use a relative path with
"SubSharedCodeRepo = ../SharedCodeRepo" (using forward slash - Unix like,
instead of backslash - windows like) and every thing works the same.

Same problem with .hgsubstate not getting updated on "hg pull -u".

I have to do a "hg revert -rtip .hgsubstate" to get the .hgsubstate in sync
with the parent of the subrepo working directory.






--
View this message in context: http://mercurial.808500.n3.nabble.com/hgsubstate-not-updated-on-pull-u-or-update-tp3995591p3995640.html
Sent from the General mailing list archive at Nabble.com.