Finding relationships between values based on their name in Python with Panda

Start with import collections (will be needed soon).

I assume that you have already read df and Fa DataFrames.

The first part of my code is to create children Series (index - parent, value - child):

isFather = df.Name.str.contains('-father', case=False)
dfChildren = df[~isFather]
key = []; val = []
for fath in df[isFather].Name:
    prefix = fath.split('-')[0]
    for child in dfChildren[dfChildren.Name.str.startswith(prefix)].Name:
        key.append(prefix)
        val.append(child)
children = pd.Series(val, index=key)

Print children to see the result.

The second part is to create the actual result, starting from each starting points in Fa:

nodes = collections.deque()
father = []; baby = []  # Containers for source data
# Loop for each starting point
for startNode in Fa.Name.str.split('-', expand=True)[0]:
    nodes.append(startNode)
    while nodes:
        node = nodes.popleft()  # Take node name from the queue
        # Children of this node
        myChildren = children[children.index == node]
        # Process children (ind - father, val - child)
        for ind, val in myChildren.items():
            parts = val.split('-')  # Parts of child name
            # Child "actual" name (if exists)
            val_2 = parts[2] if len(parts) >= 3 else ''
            if val_2 not in father:  # val_2 not "visited" before
                # Add father / child name to containers
                father.append(ind)
                baby.append(val)
                if len(val_2) > 0:
                    nodes.append(val_2)  # Add to the queue, to be processe later
        # Drop rows for "node" from "children" (if any exists)
        if (children.index == node).sum() > 0:
            children.drop(node, inplace=True)
# Convert to a DataFrame
result = pd.DataFrame({'Father': father, 'Baby': baby})
result.Father += '-father'    # Add "-father" to "bare" names

I added -father with lower case "f", but I think this is not much significant detail.

The result, for your data sample, is:

        Father           Baby
0   A03-father  A03-SA-A02-SA
1   A03-father  A03-SA-A05-SA
2   A03-father  A03-SA-A17-SA
3   A02-father  A02-SA-A04-SA
4   A05-father         A05-NA
5   A17-father  A17-SA-A18-SA
6   A04-father  A04-SA-A09-SA
7   A09-father  A09-SA-A20-SA
8   B02-father  B02-SA-B04-SA
9   B04-father  B04-SA-B06-SA
10  B06-father         B06-NA

And two remarks concerning your data sample:

  • You wrote B04-SA-B02-SA with capital O (a letter) instead of 0 (zero). I corrected it in my source data.
  • Row A02-father A02-SA-A04-SA in your expected result is doubled. I assume it should occur only once.

Commented inline

def find(data, from_pos=0):
  fathers = {}
  skip = []

  for x in data[from_pos:]:
    tks = x.split("-")

    # Is it father ?
    if tks[1].lower() == "father":
      fathers[tks[0]] = x
    else:    
      if tks[0] in fathers and tks[-2] not in skip:
        print (fathers[tks[0]], x)
        # Skip this father appearing as child later
        skip.append(tks[0])

Testcase:

data = [
'A0-Father', 
'A0-N-A2-M',
'A0-N-A4-M',
'A2-Father',
'A2-M-A0-N',
'A2-N-A8-M',
'A8-father',
'A8-M-A11-N',
'A8-M-A2-N']

find(data, from_pos=0)

Output:

A0-Father A0-N-A2-M
A0-Father A0-N-A4-M
A2-Father A2-N-A8-M
A8-father A8-M-A11-N

Edit 1:

Start with some data for testing

data = [
'A02-father',
'A03-father',
'A04-father',
'A05-father',
'A07-father',
'A08-father',
'A09-father',
'A17-father',
'A18-father',
'A20-father',
'A02-SA-A03-SA',
'A02-SA-A04-SA',
'A03-SA-A02-SA',
'A03-SA-A05-SA',
'A03-SA-A17-SA',
'A04-SA-A02-SA',
'A04-SA-A09-SA',
'A05-SA-A03-SA',
'A09-SA-A04-SA',
'A09-SA-A20-SA',
'A17-SA-A03-SA',
'A17-SA-A18-SA',
'A18-SA-A17-SA',
'A20-SA-A09-SA',
'A05-NA',
]

father = [
  'A03-father',
]

First let us make a data structure so that manipulations will be easy and lookups for relationships will be fast as you have huge data

def make_data_structure(data):
    all_fathers, all_relations = {}, {}
    for x in data:
        tks = x.split("-")

        if tks[1].lower() == "father":
            all_fathers[tks[0]] = x
        else:
            if len(tks) == 2:
                tks.extend(['NA', 'NA'])

            if tks[0] in all_relations:
                all_relations[tks[0]][0].append(tks[-2])
                all_relations[tks[0]][1].append(x)
            else:
                all_relations[tks[0]] =[[tks[-2]], [x]]
            
    return all_fathers, all_relations

all_fathers, all_relations = make_data_structure(data)
all_fathers, all_relations

Output:

{'A02': 'A02-father',
  'A03': 'A03-father',
  'A04': 'A04-father',
  'A05': 'A05-father',
  'A07': 'A07-father',
  'A08': 'A08-father',
  'A09': 'A09-father',
  'A17': 'A17-father',
  'A18': 'A18-father',
  'A20': 'A20-father'},

{'A02': [['A03', 'A04'], ['A02-SA-A03-SA', 'A02-SA-A04-SA']],
 'A03': [['A02', 'A05', 'A17'],
  ['A03-SA-A02-SA', 'A03-SA-A05-SA', 'A03-SA-A17-SA']],
 'A04': [['A02', 'A09'], ['A04-SA-A02-SA', 'A04-SA-A09-SA']],
 'A05': [['A03', 'NA'], ['A05-SA-A03-SA', 'A05-NA']],
 'A09': [['A04', 'A20'], ['A09-SA-A04-SA', 'A09-SA-A20-SA']],
 'A17': [['A03', 'A18'], ['A17-SA-A03-SA', 'A17-SA-A18-SA']],
 'A18': [['A17'], ['A18-SA-A17-SA']],
 'A20': [['A09'], ['A20-SA-A09-SA']]}

As you can see all_fathers holds all the parents and most imporantly all_relations hold the father-child relationship which can be indexed using the father for faster lookups.

How lets do the actual parsing of the relationships

def find(all_fathers, all_relations, from_father):    
    fathers = [from_father]
    skip = []

    while True:
        if len(fathers) == 0:
            break

        current_father = fathers[0]
        fathers = fathers[1:]  

        for i in range(len(all_relations[current_father][0])):
            if not all_relations[current_father][0][i] in skip:
                print (all_fathers[current_father], all_relations[current_father][1][i])            
                if all_relations[current_father][0][i] != 'NA':
                    fathers.append(all_relations[current_father][0][i])            
        skip.append(current_father)    


for x in father:    
    find(all_fathers, all_relations, x.split("-")[0])

Output:

A03-father A03-SA-A02-SA
A03-father A03-SA-A05-SA
A03-father A03-SA-A17-SA
A02-father A02-SA-A04-SA
A05-father A05-NA
A17-father A17-SA-A18-SA
A04-father A04-SA-A09-SA
A09-father A09-SA-A20-SA

Edit 2:

New test cases; [You will have to load the values in father.csv to a list called father].

data = [
'A02-father',
'A03-father',
'A04-father',
'A05-father',
'A07-father',
'A08-father',
'A09-father',
'A17-father',
'A18-father',
'A20-father',
'A02-SA-A03-SA',
'A02-SA-A04-SA',
'A03-SA-A02-SA',
'A03-SA-A05-SA',
'A03-SA-A17-SA',
'A04-SA-A02-SA',
'A04-SA-A09-SA',
'A05-SA-A03-SA',
'A09-SA-A04-SA',
'A09-SA-A20-SA',
'A17-SA-A03-SA',
'A17-SA-A18-SA',
'A18-SA-A17-SA',
'A20-SA-A09-SA',
'A05-NA',
'B02-Father',
'B04-Father',
'B06-Father',
'B02-SA-B04-SA',
'B04-SA-B02-SA',
'B04-SA-B06-SA',
'B06-SA-B04-SA',
'B06-NA',
]

father = [
  'A03-father',
   'B02-father'
]

for x in father:
    find(all_fathers, all_relations, x.split("-")[0])

Output:

A03-father A03-SA-A02-SA
A03-father A03-SA-A05-SA
A03-father A03-SA-A17-SA
A02-father A02-SA-A04-SA
A05-father A05-NA
A17-father A17-SA-A18-SA
A04-father A04-SA-A09-SA
A09-father A09-SA-A20-SA
B02-Father B02-SA-B04-SA
B04-Father B04-SA-B06-SA
B06-Father B06-NA

Tags:

Python

Pandas

Regex

Python 3.X

Numpy

Related

Using ++ as a prefix to a statement of access through class member not causing an error Apply a Function to every element in a list Fast and efficient character DataFrame creation in Rcpp How to detect if an iOS app is running on an M1 mac? Parametrized types in Raku, how to use run time values as parameters Why rust ignore lifetime checks on &str? TD with rowspan has more height than content (nested table): 1px diff in border placement of nested tables mimic the same row height of outer table Android studio updated and code too large now How to wrap @Column annotation with my own annotation in Java or Kotlin Why does calling useState's setter with the same value subsequently trigger a component update even if the old state equals the new state? Xamarin, how to generate constants for images file names in the project? System.Text.Json Field Serialization in .NET 5 not shown in Swashbuckle API Definition

Recent Posts